Yes, it's possible some of the seals designed to withstand water and glycol could get damaged
I am thinking you are creating a fictitious scenario here, so I'll roll with it.
If you filled your radiator with oil and started your car and let it run for awhile I would be most concerned with damage to seals that were designed to withstand water and glycol.
Petroleum based products could damage these seals by penetrating and degrading them. I would imagine that some manufacturers are using material that would not be effected by petroleum products and some are not.
This is assumptive on my part and I don't have any citations to back my concern.
Aside from the possibility it might damage some seals, I don't see engine oil doing damage to your cooling system by simply being in the radiator.