I'm not an automotive engineer, but the differences are primarily:
- higher rpm
- higher compression ratio
- probably a better cylinder head design (valves, ports etc.)
All these things create more power for the same reason: they result in more air being displaced in a given space of time.
The other details - intake design, lifters, VANOS, injectors, ECU maps etc. all exist only to support these features.
A few things tend to limit rpm, but I believe the big ones are valve timing and intake resonance.
Valve timing as you mentioned is controlled by VANOS, but it's not the same in the M3. The base models use single VANOS (intake cam only), but the M cars use double VANOS (both cams). (I believe M VANOS operates at higher pressure so that it can react faster, although that's really a features for responsiveness rather than peak power.)
The intake manifold will always tend to resonate at one particular frequency (which translates to one particular engine speed, i.e. rpm), and this affects the volumetric efficiency of the engine. Think of it as waves interfering with each other; when they are perfectly in phase, then tend to strengthen each other (so more air gets in). When the waves are out of phase, then diminish each other, and less air gets in. Most cars are designed with fairly long intake runners, which is optimized for low to medium rpm torque, since most cars have cam profiles that are optimized for that rpm range. Some cars actually have variable intake geometry to compliment their variable valve timing. The M3 has no intake runners at all - instead each cylinder has it's own throttle body, so it's optimized for very high rpm.
Similarly, the lifters are the older, non-hydraulic type to support the goal of high rpm - hydraulic lifters are heavy and so have too much inertia.
Finally, valve ports are typically a bottleneck on engine performance, because all the air has to go through a fairly small hole. Ordinary cars don't get perfectly optimized heads, because it would be too expensive, and would probably just move the bottleneck to the intake or the cam. But performance cars tend to get very carefully optimized cylinder heads because it's worth the effort, when you have everything else in place to get high air flow.
Once you have your power goal, that tells you how much air you need to move. Then you can decide whether to achieve this goal via displacement, higher rpm, or forced induction, or some combination of these. Once you decide on that, the rest of the details - block material, mass of reciprocating parts, valve timing, intake design etc. all follow from that.