First of all, realize the handbrake in and of itself is used for holding the vehicle, not stopping the vehicle. You could possibly use it in an emergency if so needed, but that's most likely a once in a lifetime ordeal. If it's a dedicated hand brake, there's very little friction material or surface area on the drum/disk to actually do the job. If it's a shared hand brake (one where it's used for braking under normal use and for holding as for the handbrake), there only a little bit more friction material or surface area. Under normal use, the rear brake pads are used for about 30-35% of the braking for the entire vehicle (the fronts do the rest). This is the reason they are smaller without much friction materials. To continually use the handbrake will cause a huge amount of wear and will destroy the friction material it uses well before its time, because it's getting used for something it wasn't designed to do (ie: stopping the vehicle). This applies to either handbrake solution.
How much damage will occur? It isn't that it will cause damage, per se, but rather it will wear the braking items out much faster by doing this. The factors involved are: how much it's used like this; how big the car is; how fast are you going when you pull the handle; what type of friction materials are involved; etc; etc; etc. There is no equation which is going to give you an "if I do this 20 times, my brakes will be shot" type of output. There's just no way of telling. Just know, it's going to happen much quicker. Realize, since a dedicated handbrake sees very little frictional wear ever, any abuse of it is going to cause severe wear. There's just no way around it.
The big thing you need to be made aware of is in most of the situations (if not all) you pointed out in your question don't use a parking brake to do the deed. They have a separate braking caliper along with a braking handle which allows them to operate either side independently, then still have normal braking besides. A system could look something like the following image:
Plus they use a braking mechanism which could look something like this:
When you push on the handle it applies brakes to one side of the car, while when you pull the handle it applies brakes to the other side. In other words, they never use the emergency brake at all, which is the typical way drifters and the like will manage their braking needs.