Having to wiggle the turn signal lever to get hi beams to go off sounds like there is something wonky in the turn signal lever.
There was a problem with some early 03 Matrixes, and to a lesser extent some 03 Vibes where the turn signal lever could cause the headlights to come on randomly at night when cold. Toyota put out a TSB on it,
https://www.tsbsearch.com/Toyota/EL003-03 but it doesn't include the procedure to replace the turn signal lever. This record breaking summer makes it unlikely your issue is cold related, but after 20 years, definitely possible your switch is suspect.
That TSB gives the Toyota part number as 84140-02200, and there's lots of them on ebay, a number are sold as "new" for around $25, I wouldn't go for a used one.
I couldn't find a procedure specific to Vibe/Matrix, but I found a youtube for a right hand drive corolla, and if Vibe/Matrix is similar, look like you don't have to pull the airbag or steering wheel, which would be really nice to not have to. I got a kick out of the video where the tech struggles to get the cover trim off, and to undo the electrical connector. That's the kicker in reading a procedure, it will list a simple step like "Disconnect the connector" and you have to spend seemingly endless time trying to figure out exactly how to do that.
edit, found a better video at
https://www.youtube.com/watch?v=C7UV8bkDOdg
Anyway, good luck, hope its an easy fix. Let us know how it goes