It's difficult to reach a conclusive answer, but you're on the right track with the extra mass of a RWD transmission - the propeller shaft taking power to the rear wheels adds some considerable weight. The gearbox is also traditionally attached directly behind the engine, and this means it impinges on the passenger compartment. A tunnel has to be constructed over both the gearbox and prop shaft, which requires more metal versus a flat passenger compartment floor.
More moving parts also means more bearings, which means more sources of heat loss. Compared to FWD, there are bearings at either end of the prop shaft, and generally (on cars with two-piece shafts) a 'carrier' bearing in the middle. Some bearing designs require periodic lubrication, which is easy to neglect.
The differential on a RWD vehicle also has to rotate the drive through 90 degrees - the rear drive shafts are perpendicular to the prop shaft. Some energy is lost as heat in the differential doing this. By contrast, FWD cars use a more linear differential design (usually built directly into the gearbox) which has most of the gears rotating along the same axis.
AWD vehicles is much of the same - extra weight (second differential, second pair of drive shafts, prop shaft) and extra moving parts needing more bearings. Some designs use a simple mechanical central differential, others (such as Subaru) use a stack of clutches and vary the slip to transfer power between front and rear. Any time two components are not being driven in sync (such as clutch slip), there will be energy loss through heat.
In general, all the extra energy is being lost as heat, either through friction in the drive components, or through burning more fuel to compensate for the weight addition.