The 4-cyl camry's of that vintage are known to have a rough idle, but a drop to 300-400 RPM at idle is pretty severe. I haven't seen the spec, but I would think that car should probably warm idle in the 800-900 RPM range.
You should have the car scanned to check for pending trouble codes. These codes can show up before the Check Engine Light (CEL) goes on, and they can be a huge diagnostic clue as to what's wrong. I'd expect to find something here if your problem is as severe as it sounds--the computer should have logged some misfires.
The generally accepted stuff to check for a "normal" rough idle would be
- intake system: clean the throttle body and mass airflow sensor, check for vacuum leaks;
- ignition system: spark plugs, wires, and ignition coil(s);
- fuel system: injectors, filter;
In your case, I'd suspect something like a bad sensor or bad ignition coil--something that can really screw with the system profoundly.