Welcome! The 03/04 nose, as far as I know, should fit right up to an 05+. It's a two-piece bumper, but that doesn't affect mounting. I would be surprised if it hasn't been done before.
Just curious as to why you're wanting to swap from an 03-04 to the 05-08 nose You may find a member here @ GenVibe willing to trade you even up. I know if I had the 05-08 nose I'd be willing to trade for the 03-04 as it looks sportier. Others in the past mentioned doing such a thing but it's been a while since I've read anything like this. Good luck!
I currently have an 05 and, while I love her dearly, I think the front looks just a little too big (like a fat lip). Seen some other pictures of 03/04 Vibes that are the same color and their front looks a lot better (IMHO).