Once again, the search function on this forum saves the day...

Some XFs have their window switch functions reversed -- push down and the window goes up; push up and the window goes down.

I was frustrated by this tonight and just pulled the shift boot out, yanked the plastic cover holding the switches, and reversed the switches. It was so easy, and now my windows operate "normally."

I got home and checked the forum through a search and found a thread from 2004 when some other owners figured this out:
Reverse Window Switches

In brief, pull the aforementioned panel (no tools needed), and release the two clips holding each switch. Turn them around (180) and snap back in. Replace panel and shift boot (for manual trans) and you're all set!

I thought I'd resurrect this thread five years later for fellow, frustrated XF drivers.

Again, some XFs apparently don't have this problem, and others have drivers "conditioned" to the backwards switches!