A lot of that depends on the type of vehicle, and your definition of 'fast'.
Take this with a grain of salt, but a while back BBC's Top Gear did a test involving a V8 powered BMW M3 and a Toyota Prius. The Prius was driven as fast as it could go around their test track, while all the M3 had to do was keep up. After a few laps, the Prius had done ~18 mpg, while the M3 achieved ~20.
There are several factors that could have caused this:
Obviously, the M3 is much faster than the Prius in a straight line, so while the Prius was buzzing it's little 4-banger engine to redline, the M3 could trundle along at half throttle, short-shifting, and never having to use more than a small fraction of its power.
When the Prius approached a corner, it slammed on the brakes, scrubbing all the speed it worked so hard to achieve, so it could round the corner on its tiny, low grip eco-tires. The M3, on the other had, could coast into the corners, and retain much more speed due to its large, grippy, performance oriented rubber.
The suspension setup also provided the M3 with better cornering performance, while the soft suspension on the Prius was more comfort oriented. Same goes for weight distribution/center of gravity, aerodynamics etc.
Using the brakes has a direct, negative effect on MPGs, as you are essentially turning the gas you just used to speed up into heat energy in the brakes. In everyday driving, this generally lends benefit to slower drivers. When that light you were speeding toward turns yellow, you have to slam the middle pedal and waste all that gas you used getting to 50 mph, while grandma back there had only accelerated to 15 before coasting to a stop right next to you.