TL;DR
It's not possible to prepare a Virago 535 so that both short and tall riders can ride it comfortably.
Full answer
After investigating a lot into this issue, and even after trying a few other motorcycles, I concluded that it's not possible to prepare the small Virago so that both me and my wife can ride it comfortably. The size and ergonomics of the motorbike are perfect for a short person, and even if I could get it more comfortable for me, it would become less comfortable for my wife.
At first I considered switching handlebars, then seats and after a while I was looking at suspension and how to alter the motorbike's geometry (although I was definitely not going to do that). I realized that there was nothing I can do make the bike great for both of us. Every little thing I changed to make the motorbike more comfortable for me, it would make it less comfortable for my wife.
We tried different bikes to see what we could manage (Dragstar/V-star, Vulcan, Intruder, Shadow, ...). All bikes were larger so they were better for me, but they would be too large for my wife. In some of them, she couldn't even reach the foot controls.
The only bike we could actually manage to drive was the 883, but still not perfect to both (as expected).
I ended up buying a new motorbike for me (Volusia) which is a bit too large for my wife to drive, but she keeps the Virago and we're both happy. In the end, I guess that if two people are too different physically, they will need to have two different motorbikes.