What do you do when brand new pads are rubbing on new rotor? I changed both rear rotors and brake pads but passenger side piston was so hard to retract. Had to rent a heavy duty tool because mine kept spinning. Well took It on a 5 mile drive It was smoking and smelled so bad. I looked like it is going to catch on fire. Does this mean the caliper is bad?
It does sound like the caliper is not releasing, which means it is bad, or at least needs some repair. You might have corrosion on the slides, or the caliper pistons. The caliper needs to be replaced or repaired, and you will likely need to replace the pad as well.
High heat in the brake caliper can cause the brake fluid to degrade, or develop air bubbles. You should bleed that brake as well.
In a properly functioning brake system the pistons in calipers must be free to move (in both directions). When brakes are applied, the fluid is under very high pressure thus generally the pistons will extend when brakes are applied given enough force on the brake pedal, but when calipers are sticking the much lower force of the rotor wobbling between brake pads won't be sufficient to push the pistons back into the calipers for the tiny gap required when brakes are not in use. When this happens, brakes smoke (and wear at an abnormally high rate).
Had to rent a heavy duty tool because mine kept spinning.
If the normal tool for retracting the pistons failed to move one or both of your pistons, the calipers aren't operating properly. Even if a heavier duty tool managed to retract the calipers enough for you to be able to fit in the new brake pads, this tool doesn't help when you are driving and the pistons must release the pads from the force of the rotor wobbling.
Get new or rebuilt calipers or rebuild them yourself. Rear calipers tend to cost more than the front calipers because they have more parts, which makes DIY rebuilds save more money compared to rebuilding front calipers but also more complex. For street cars, using e.g. RockAuto to source rebuilt calipers is a very sensible choice.
For the most part, D. SM is generally correct. If you had to rent a heavy duty tool to compress the caliper pistons then you should be looking at rockauto.com for either the proper rebuild kit or rebuilt calipers (it's a mater of time vs money). From what you've described it seems as though the caliper pistons are seizing up and although you were able to compress them to put on the pads, once you use the brakes it will return those caliper piston(s) to an extended state pressing the new pads against the rotor allowing friction to do it's thing and heat them up which is what you were smelling. Check that suspect rotor and you may see some "bluing" (rainbow-esque coloring on the rotors) where they (the rotors) have been heated up beyond their heat range. Get that taken care of immediately as new calipers are cheaper than an insurance claim or a lawyer.
One thing I'd like to point out is that rotors don't wobble, and if they do, you'll feel it in the steering wheel (if they are up front) or in the brake pedal (if they are in the rear) where the wobbling occurs. The rotors are held in place by the wheel which is tightened down by the lug nuts. No wobble. The pads all have some sort of "return" spring to help keep them from constantly rubbing on the rotors for the most part.