FYI -- I just replaced my scratched 2005 silver shifter bezel with one from a 2004. To dull the glossy finish, I used Turtle Wax Rubbing Compound, which is like a super-fine wet sanding. Worked well, and the resulting color matches.

Because there were a lot of 2004 XFs made, I suspect many other 2005+ owners replacing interior parts with used ones are running into this minor problem too.