I drive very carefully and efficiently, accelerating slowly and using the EV mode wherever I can. But I can only drive so carefully before I start getting honked at by other folks for driving too slowly.
I think that's your problem. You should drive like an average person, because that's what the car is optimized and tested for.
As an example, you would think that driving slowly saves fuel. But if you drive the slowest possible, 0, that is the most inefficient condition, because the engine is idling and you are going nowhere.
The efficiency of an internal combustion engine and drivetrain peaks at some point, for which engineers have optimized. That point is usually cruise conditions. Low powers are inefficient, and so are high powers. (This is demonstrated with the Prius vs BMW M3 efficiency as others have mentioned; they are optimized at different points.)
There are various technical reasons. For example, when an engine is going at too low of an RPM, the gases stay too long in the cylinder before being expanded to do work, and heat is lost through the cylinder walls.
The idea of a hybrid is to widen the efficiency peak by bringing the system closer to the optimal point at all times. For example, if the engine is producing less power than optimal, we can store some for later. If the engine needs to run at a higher level than optimal, we use stored energy to relieve some load.
What you should do in most cases is to get the car to cruise conditions quickly and keep it there. You should accelerate at an average to slightly aggressive rate to cruise speed, then back off and keep it at a constant speed as long as possible.
Where you should be gentle is deceleration. Slow naturally as much as possible, by rolling, or at a rate that does not require mechanical brakes.