I drive a 45 year old landrover, and do this frequently off-road and occasionally on-road.
Reading into your question, you only change out of gear into neutral. If you're coasting toward a red light, the road wheels are driving the engine, the engine is resisting that load, and you are on the "overrun" or engine-braking. The wheels are pushing the engine through the gearbox.
To change out of gear smoothly you want to either
- blip the throttle/accelerator/gas pedal just enough you make the engine speed equal to the velocity of the vehicle.
- throttle off / raise the right foot and slip the gearstick just as the vehicle's acceleration hits zero.
Either way, this means the teeth in the gearbox are not pressing hard against either side. In that short sub-second window you can slip the gearstick into neutral and coast up to the lights with less resistance, and less wear. Your engine is idling instead of running faster.
tl;dr: minimal to no extra wear if you change smoothly
Downside 1: If there's an impending accident and you suddenly want a burst of power, you'll have to engage a suitable gear quickly. Get a bad gear and you could stall instead of moving.
Note this is different to pressing the clutch pedal and coasting in gear - a manual clutch is more fragile when apart and its possible to destroy clutch plates by spinning them too fast with the pedal pressed down.
With the exception of pulling away from a standstill, I can change gear up and down by waiting till the engine revs match the roadwheels, and slide it right in or out. This is not a racing change, its all about timing and matching the road noise/vibrations to the engine note.
A light finger pressure against the gearstick helps read the vibrations. No sudden changes of throttle till the gearstick is fully home else you increase the risk of chipping teeth.
As long as you drive gently and change smoothly, the wear would be the same as changes with the clutch. A loud change with lots of graunching noises is worse.
Another advantage: Its generally ill-advised to get your clutch plates wet when offroad so changing gear while in water deeper than your bellhousing is bad.