Using them all year round, is there a downside to them? Or is it just a matter of costs. I will refrain from further widening the questions scope.
Yes, there is a downside.
Some winter tires are studded. They cause terrible air quality and pavement wear issues. They literally kill people, because of the poor air quality caused by them, but at the same time, cause less traffic accidents and save people. Needless to say, there is no reason to use such tires in summer.
It is necessary that a fraction of drivers use studded tires, because they modify the smooth ice on roads to be rough, so that non-studded winter tires can obtain useful grip. So, 100% studded => air quality and pavement wear issues, least amount of accidents, 50% studded / 50% non-studded => half of air quality and pavement wear issues, still acceptably low accident count, 100% non-studded => no air quality and excess pavement wear issues, but a terribly high accident count.
The main question is, should you use non-studded winter tires in the summer then? The answer is no. Due to the different rubber compound, they have poorer than optimal traction and wear extremely fast. Furthermore, the feel to the road is less direct, due to tread squirm.
At least where I live, it is legal to drive on non-studded winter tires in the summer. It is illegal to drive on studded winter tires in the summer. However, for best traction and lowest cost, it makes sense to use summer tires in the summer.