Hi Adam, welcome to the forum. Seems like there are quite a few 04's changing hands lately with many of the new members showing up. I too have an 04 that I purchased brand new, and I plan on keeping mine as long as I can.
At 6'4" you are definitly "pushing the envelope" when it comes to finding a comfortable driving position. I'm 6'2" and I only have about an inch extra for my knees, and about 1" clearance above my head.
I think removing the knee bolster (like AMX-Jim) suggested would be of great help to you.
Another forum member documentated his seat modification where he was able to get his seat to sit closer to the floor. Some cuttting was needed, but all in all it didn't seem to be that major of an operation. I can't remember who it was, but maybe somebody else will. You might be able to find his thread by doing a search too.
I'm only aware of one person cutting into the bulkhead and that was forum member "PADGETT". I think he cut a notch out of the section directly behind the drivers seat so it would lean back a few more degrees. I'm not sure how deep he went, (because the gas tank isn't too far beneath it) so you may want to PM him for more infomation before trying it yourself.
When I bought my Crossfire, I had only sat in about 4 or 5 other XF's, and I never could get comfortable. I almost gave up on buying one (even though I felt it was the perfect new car for me at the time). I wound up winning a fantastic deal on EBAY for exactly the car I wanted, only it was a nine hour drive away. When I picked it up, the salesman (who delivered it to me just outside the showroom) had the seat perfectly adjusted for me when I first got into it. (He too was 6'2"). I asked him how he did it since I was never able to find the "sweet spot", and he said, just drop the seat all the way down, and with the seat back in an up right position, slide the seat back as close to the bulkhead as possible. Then after getting in, move the seat forward until your knees are close to the knee bolster, then tilt the seat back as far as it will go. After tilting the front of the seat up a little to give the bottom of my legs just a little support, it felt perfect. In the last nine summers I've spent enjoying my Crossfire, I've spent many times driving it for 12hrs straight on trips, and never got out of it with a sore back. It did take a short break in time to get aquainted with the hard seat bottoms, and some people find removing their wallets while driving improves the trip.
I'm sure after you make a few adjustments, it will feel like home to ya. Good Luck, and enjoy.