I have a bit of an inside line on this one...
Keep in mind that there are numerous generations of Z's to add to the mix... The Z33 (Nissan-speak for 350Z) is the volume seller and is a good car, not a great one. Boosted variants DO exist but the vast majority are 13.5-14.5 cars... Driver mod, as with any car, is crucial. The Z33's horsepower is higher than the base XF, but so is the weight by some 600 pounds. Z has slight advantage in the corners due to the better suspension geometry. But, let's not forget brakes. The brakes on the Z33 and even the Z34 have been notoriously poor. The XF, on the other hand, stops in incredibly short distances.
The Z34 (370) is a quick car and should not be challenged by the N/A Crossfires. Period. Not on the straights, not in the twisties. If the driver of the Z34 sucks then you might edge him out in the bends, but once the straights open up, it's game over.
With an SRT, they are easily dispatched in the straights but the Z34 is an excellent handling car. VERY balanced in the corners, and much more predictable than the on/off power we have due to the stability control kicking in.
Personally, the engines are too harsh in ALL the Z-cars (the upper RPMs are extremely thrashy) and the refinement is lacking. Getting any Z into the low 12's takes quite a bit of work, and the 370's motor can't take a tremendous amount of prolonged abuse without a thorough workover.
The Z is the Mustang of Japan. Very popular, solid performance, volume car. Does not excel in any category, but isn't embarrassed in any either. HUGE aftermarket, and an even bigger enthusiast following. A great sports car IF you don't mind seeing one every few miles. Only a head-turner to enthusiasts, and each revision/model change is very drastic, meaning the prior generation looks very dated.
A good car, but I would never consider trading my SRT for one. Simply too many in my area