My 1998 Honda Civic LX appears to have oil sludge in radiator overflow tank. Mechanic says I need to replace engine or buy a new car. What would cause this to happen. Have been very faithful in car's maintenance over the years and am only owner.
There is an unwanted passage between your oil system and your cooling system. Probably the most common cause of that is a head gasket with a leak between the two, which could be caused by high compression/detonation (there are several possible causes for that including incorrect ignition timing, low octane gasoline), an engine overheat, uneven head bolt torques and/or a defective head gasket.
It could also be a cracked engine block. That could be caused by uneven heating/cooling (an overheat could do it), or from inadequate freeze protection resulting in frozen coolant in very cold weather, or other mechanical defect. If it is a cracked engine block, I think you really are in for an engine replacement. Even a head gasket replacement is expensive (~$1,000) on overhead cam engines, which Hondas all are.
Before you replace the head gasket or engine or car, this Steel Seal stuff might be worth a try. Here's a Scotty Kilmer video on that. I've had it help on an older BMW, the only car I tried it on.
If you want to figure out whether it really is a blown head gasket or not, here's a Scotty Kilmer video demonstrating a test kit, but if there's oil sludge in your coolant, you definitely have some problem.