cdunn's answer is spot on.
To add a bit (especially for cars), in a race car you often want to be dropping from something like 7000 rpm to maybe 5500 rpm (or an even higher band, depending on the engine) in a split second to make an upshift. Especially at higher engine speed, that's a lot of energy to bleed (and goes as a function of the weight of the flywheel).
The lighter the flywheel, as soon as you hit the clutch, the faster the engine speed will drop. This is a double-edged sword. Not only does it let you shift faster, you must shift faster - and more precisely. The time between clutch and shift (at ideal rev-match) is reduced, and the window of opportunity to make a perfect shift is also reduced. Your revs won't just hang around, slowly bobbling down (giving you lots of time to fuss with the shift lever) - they sink like a stone and you must make the shift before they drop too low (or else you get a rude shock when you let the clutch out again).
Same deal for downshifts - a heel-toe gives you a very fast RPM bump but, again, you have to make the shift in the brief window that presents itself.
Clutch springs are usually stiffer on racing clutches as well, so badly timed shifts are even harder on your vehicle - not only will a sloppy shift kill your race time, it will hurt your car as well.
The low weight of a racing flywheel also makes moving off (from a stop) harder. When you're racing, you only do this once per race - sometimes not at all if it's a rolling start. When you're commuting you do this all day long. A heavier flywheel carries much more momentum - when you feather out the clutch from a stop, this momentum helps to keep the engine running at a steady RPM as you add load and gives you more latitude in your throttle control to keep the car going and setting off smoothly.
With a racing flywheel, you get no help here - as you feather out the clutch from a stop the engine will bog down rapidly (and stall) if you don't add fuel to keep the revs up. But because the flywheel is so light, adding just a bit of fuel sends the engine racing off to high RPM if you jump the gun before the clutch starts to engage. This whole procedure is much more sensitive with a light flywheel (and a stiff, grabby clutch). Daily driving like this would be a nightmare - stop-and-go traffic would be a full body workout and mental exercise in concentration.