Here is some info I ran across which is generic P0505 info ; I cut some of it out to get to the troubleshooting part:
P0505 IAC Malfunction
OBD-II Trouble Code Technical Description
ASE Certified Technician
IAC (Idle Air Control) System Malfunction
What does that mean?
The IAC (Idle Air Control) valve controls the idle speed of the engine. At idle the throttle plate is closed, of course. The IAC is computer controlled and allows only a certain amount of air around the throttle plate. This metered air controls the idle speed. The IAC has a "plunger" or "shutter" that moves to open or close the bypass air passageway.
This should be a fairly straightforward diagnosis. Start by clearing the fault codes. Then unplug the IAC and start the engine (This may cause other IAC codes to set. Just ignore them for now). If the P0505 code doesn't reset, replace the IAC. It is internally shorting one of the IAC circuits. If the code DOES return after unplugging the IAC and starting the engine, then you know it isn't the IAC motor. Visually check the wiring harness all the way back to the PCM. You may need to visually inspect the harness by removing all the insulation off the wires. It is possible that the wires could be shorting to each other or to other wires in the harness. If there are no wiring problems, and the code keeps returning with the IAC unplugged, you can check the PCM by removing the IAC wires from the PCM connector (or clipping the wires in a location where they can be repaired easily) and then start the engine (If you're not sure, don't do it. Take it to a qualified professional). Now if the code doesn't reset there is a problem with the wiring. Check for continuity on all the IAC wires. Any resistance points to a problem. Check for voltage on any (At this point there shouldn't be any voltages on the wires because they're clipped at one end and unplugged at the IAC). Repair as necessary. But if the P0505 code DOES reset after removing the IAC wires from the PCM connector (or clipping them), replace the PCM, there's a problem internally on the IAC control circuit.
Other related DTCs: P0506, P0507
More info and watch video: https://www.autocodes.com/p0505_toyota.html
See section 05-264 of manual posted here for help with wiring pin outs: http://madstyle1972.com/Repair/
Besides the above with wiring and connection to check and with all that you have done, I would plug all vacuum ports at the engine, on the engine including the power brake booster. Start engine and see if idle is better. You are going to get other error codes from the ECM/PCM, and you can clear these later, but if the idle speed is better then you have a vacuum leak. Hook the lines back up one at a time until the idle goes wild again and you will then know what system has a vacuum leak once you trace that vacuum line down.
Is the engine sealed? Meaning no internal vacuum leaks on motor. Oil fill cap tight and sealed? Any motor oil usage? A high amount of engine blow by past the piston rings could overwhelm the PCV system and force air into the throttle body that the IAC can not control. Do you have the proper PCV valve in engine? Any vacuum leaks or un-metered air entering the engine will cause a P0505. And what all these sources say and you have not gotten to is the ECM/PCM being bad. It looks like you checked about everything else but do not guess. Verify, verify, and if all else fails, verify again that you are checking these things properly. Just because you checked it in the past does not mean it is good now. This looks like a real puzzle and the nasty ones to figure out can be the ones that have more then one component bad. Good luck and I hope you can get this nagging problem figured out.