More than likely, you have an air leak
If you haven't cleaned out and rebuilt a bunch of 4 carburetor banks in your life there is a good chance you have accidentally created an air leak in the system. Unmetered air will create the symptoms you describe.
The giveaway on most lean conditions is the falling idle. It idles high and is getting enough fuel but the cylinders with the lean condition begin to misfire, the idle rapidly drops and eventually the engine stalls.
Places to check
- Manifolds - you have four rubber manifolds that are 34 years old. The manifolds get dry rotted. The rubber goes through too many heat cycles and too much UV light exposure over that period of time and the manifolds can easily crack and allow unmetered air into the system. Unmetered air enters the engine between the head and the slide or butterfly of the carburetor or throttle body. This changes the anticipated Air Fuel Ratio and creates a lean condition.
Here is some information regarding dry rot of a tire.
Here is a QA on identifying lean or rich conditions on a motorcycle.
Here are some methods to troubleshoot and discover if you really do have a lean condition.
After you have completed your troubleshooting and resolved the issue, you will want to balance your carburetors. Here is some instruction on that procedure.
It's frustrating issue to troubleshoot, patience is the key as well as tenacity.
I was thinking this may be a valve adjustment issue but your post doesn't have all the symptoms. If your clearance is too tight and you need to put in thinner shims you will get similar symptoms but the bike won't start when it gets hot after a ride. You would need to wait for it to cool, the valve stems shrink just enough to allow the valve to close all the way and then you can start and it acts normal until it gets warm/hot. So I believe that focusing on a lean condition, checking all of the vacuum lines and ensuring they are connected to your vacuum petcock, your vacuum fuel pump (yes, you have one of those on that model) as well as the YICS bolts on the sides of your cylinder head along with the manifolds is going to be key to success.
Check those links in this post. The information you need is in there.
For good measure, here's a link to instruction on adjusting the valves on a very similar bike to yours. In fact, it's the same exact cylinder head. Good luck.