When replacing my car's coolant I overtightened one of the air relief plugs and stripped the internal threads. It's located on the inlet manifold. The torque setting for the plugs is only 61-69 in-lb so it doesn't take much to strip the threads. I assume I will need to tap new threads but I'm unsure of how exactly to do it. Do I need to drill the hole out first?
Depends on the material and amount of material available.
It can be possible to re-tap the existing thread but perhaps not in your case.
Another possibility is to fit a tapered plug if you can get one in a suitable size.
Then taking it out to the next plug size is possible as long as there is enough material, but this usually means drilling out and then tapping from scratch with a starting tap and then parallel tap.
A different idea from tapping the hole to a larger one would be to repair the threads which are there already using an epoxy like JBWeld or the like. You could either just epoxy the part in place if it really doesn't need to be removed again, or if needed, there are other methods/products you could use which would create new threads where the fitting could be screwed into. These types of products are still an epoxy (like JBWeld), but comes with a thread release agent which would allow the fitting to be removed after the epoxy has set up. Since the part does not require a lot of torque, using something like this is a lot less destructive than tapping a new hole in a place where you don't have much strength in the first place.