I also drove them both on the same day.
The RX-8 was much stiffer, and way too noisy. As people have said, it depends on what you like.
I bought the Crossfire because (a) it's much better for every-day driving and long-distance driving. Overall it's much more luxurious. (b) the Crossfire might have a chance in the snow (I live in New England). RX-8 would have none.
You might get a different feel for this if you also checked out the RX-8 forum.