What is the difference between high octane and low octane fuel? I'm not asking if one is better than the other for my vehicle, specifically, but rather what occurs differently both scientifically and mechanically when either is used?
So-called "octane number" is a measure of how much the air-fuel mix is resistant to detonation.
The higher the "octane number" is the more resistant it is and the more is can be compressed without exploding. The higher the pressure is at the point the air-fuel mix ignites the more efficient the engine works.
Modern engines are designed for some rather high air-fuel mix pressure, if the mix ignites before that pressure is reached the engine just won't work right, because ignition will happen at the wrong moment of the engine cycle. That's why only fuel with right "octane number" should be used for any engine - depending on how that engine is designed, otherwise it just won't work right. The vehicle manual usually specifies which "octane number" is right for that vehicle engine.
Here's the deal : higher octane means less volatility, and this is meant for engines with higher compression so that the fuel doesn't ignite too early in the cycle . Lower octane fuel is more volatile and is meant for engines where compression rates are lower, so that the ignition in the cycle not occur too early .
In all cases, follow the manufacturer's octane recommendation for better performance . However most cars which use high octane are able to accept regular gas because they are equipped with knock sensors which will automatically adjust the engine cycle timing, thus reducing engine knock or detonation. Engine performance will be diminished however.
For an engine, there is a specific peak pressure that it can withstand, which is limited my the mechanical stress limit of the components of the engine parts. This peak pressure is in direct relation to the compression ratio of the engine and the octane rating of the fuel. The compression ratio of the engine stays constant. With a low Octane rating, there will be misfires happening at an undesired position of the piston, leading to drop in the power extraction from that stroke, and undesirable detonation points in the combustion chamber which leads to melting and cracking of cylinder head and engine liners. Too much octane rating makes the mixture more resistant to compression, leading to the escape of much of unburnt gases to escape during the exhaust stroke leading to drop in the efficiency. So for any engine there will be a specific octane rating mentioned by the manufacturer and it should be used for maximum efficiency.
The octane rating tells you how resistant fuel is to premature detonation, i.e. burning before it's meant to. It's quite a long explanation, depending on how much you already know. The short answer is that high octane fuel allows you to fire the spark plug closer to the optimal point during the compression stroke to allow the engine to perform at its best. I found this page, which covers the topic pretty well and should answer your question thoroughly: