I'm not a fan of the 350Z, either. If you want to buy a car for speed ... which, to an extent I find silly ... then go for the 350Z. But there will always be faster cars, and as such you'll be continually disappointed.
Go for the car you like the most. The Crossfire is a far more luxurious and refined ride. It's easier to be in, it's quieter, it's softer. The 350Z is harsher. My vote is for the Crossfire, but be sure you testdrive both.