If you had a perfectly-straight set of wheels, you'd find they would still swerve off to a certain side after a while. That's why you have to make micro-adjustments on the steering wheel. If they were perfectly straight, you'd A) struggle to turn the wheel if on an un-assisted steering rack and B) you'd be making more frequent and more prominent adjustments with the steering wheel whilst driving.
If you ever had one of those little toy cars that you could pull back, let go, and then have it launch off, you'd notice they would never go in a straight line (which always annoyed me because I loved trying to get it to go between little lego buildings I used to build as a kid) and would find itself somewhere else to where I wanted. You'll also notice that if you drive down a road with a 'pitch', where the centreline is ever-so-slightly higher than the edges of the road, your steering wheel tends to swing to the side of the road. Yes, this is normal, but if you're holding onto the wheel, it's almost unnoticeable. If you had perfectly straight wheels, you're going to be physically fighting the wheel to stay straight when you're going over 15-20mph.
No, I don't have any technical data to back up this, but I've had plenty of time experimenting with cars to find different set-ups, and also helps me diagnose some problems with steering when you've seen what effects are caused by what firsthand.