Luckily, I drove slowly, about 20-40km/h. When I reached home, I realised that I did not release my handbrake. Will there be a big problem?
If the handbrake was on well enough to stop the rotation of a rear tire(s), you may have created a flat spot on them due to wear. If the tires were able to spin, either adjustment of the brakes need to occur, or possibly replacement of friction material may be needed. If enough heat or wear was generated, replacement of the rotor/drum may also be required.
Your rear brake should be pretty hot, but I can't see why anything permanent would have happened.
One time I drove home after having my brakes inspected, only to realise the idiot had adjusted my rear brakes in such a way that they never disengaged. I could feel the heat radiating from the rear-left drum. I waited for it to cool, then took it apart and readjusted. Three years later I've still not had a problem.
When I was young my mother drove from Vegas to Orange County CA with the handbrake engaged. As far as I know it never caused a problem, although she may have had it looked at later without me knowing--I was pretty young.
In general I'd say no problem if the handbrake still works but it wouldn't hurt to have a look for wear.