From my understanding, the primary concern is the frame itself. If the frame is bent or damaged, it can alter all sorts of vehicle geometry, and even small changes can have a big effect. A simple example would be a bent radiator support. If this wasn't properly repaired/replaced, then the radiator would not cool as needed and could lead to further engine damage. Likewise, if you had steering misalignment that could cause worsened handling and excessive tire wear.
Another aspect (that will vary depending on country and laws of a given jurisdiction), is that you don't have the right or ability in many cases to chase up the repairmen for faulty work. Meaning, that if a person got in a accident and had a new transmission installed, then you bought the car and agreed to "as-is" and the transmission went bad after 100 miles - you'd have no ability to leverage the company to fix whatever it messed up!
Basically, when a car is in an accident, there are A LOT of things that can get messed up, and it's near impossible to be 100% certain that none of those things are perfectly fine without fully tearing the vehicle apart and measuring everything against factory spec (which, needless to say, no one wants to spend the time doing). That is a very time intensive and therefore costly endeavor.