Yes and you should. Rust is always bad and it sounds like you have a relatively easy job. Remember, if you have rust, you don't have paint. If you're careful, there shouldn't be any effect on the paint surrounding the threaded hole.
In terms of cleaning the threads out, you're going to need an abrasive to remove the rust and then a tool to remove any debris. If you'd like to start gently, I'd recommend something like a white toothpaste (not gel!) applied to the threads of the screw. Work the screw in and out a few times to grind your minty freshness into the threads. Now, if you spin a thin pipecleaner in the threaded hole, you should be able to get out a bunch of dirty toothpaste along with some portion of the rust. Repeat as necessary.
Once you feel that you've removed the rust that's going to come off, I'd recommend switching to vinyl plate screws. They hold just fine and certainly won't rust on you.