I love driving with the drivers side window down because I enjoy the sound of the Crossfire's motor so much. (I hardly ever listen to the radio).
When the wind buffeting starts, I just crack open the passenger side window (less than an inch) and that seems to take care of it.
As Andrew pointed out, this is the worst car that I've owned for this problem.
Still, it's a small price to pay when you consider all the other advantages.