for questions like this, I often go to car-parts.com, a database of junkyard inventory. They use some authoritative interchangeability reference, and when I searched for a 2005 Vibe front driver seat belt/retractor, it offered me results from 2004-2008 Vibes. When I searched for one from a 2003 Vibe, it only offered me others from only 2003 Vibes.
similar for Matrix, 2003's apparently were different, and for some reason, that site's results imply that there is some difference between vibes and Matrix/corollas, although I don't understand why they would be.
I would have assumed that all the 03-08s would be the same, but given the search results from that site, I wouldn't risk dropping a 2003 into a 2005, just to be safe, which is the whole point of your question.