The bottom line is yes, if you drag your exhaust over the speed bumps, it will cause damage to your car. It will be mainly localized to the exhaust on the vehicle. Besides flattening out the exhaust pipes, you also put stress on the joints and the hangers for the different mounting points. If you hit the exhaust hard enough, you can cause further damage to the under side where the exhaust may be pushed up into other parts. If hit hard enough, it can also force damage up into the exhaust manifolds on the engine.
The damage to the speed bump could be caused from any of several reasons. It happens when the lowest part of the car is not high enough to go over the speed bump. It might be the exhaust which hits (most likely part on modified cars). It could also be other parts. For instance, I had an aftermarket torque arm on my 94 Camaro Z28. The mount for it hung low due to it's design which allowed the single exhaust pipe to go through without touching it. It was in the center of the vehicle. Since it hung low, then add in 1.5" worth of drop from lowering springs, this made it really tough to go over speed bumps without hitting anything.