I'd say trade for an SRT-6 if (and only if) you crave better straight-line performance. Otherwise, in my humble opinion, the NA crossfire is just as good as the SRT-6 on public roads at sane speeds.

Aside from the engine, I don't see the SRT-6 having much of an advantage over the NA models. You get much nicer seats... and very firm suspension which could be a plus or minus depending on how you'll be driving. For me, the suspension can be jarring and annoying over rough roads, but I appreciate the stiffness when out in the twisties (maybe 10% of my driving at most).

If driving a nice, decently fast sports car makes you happy, I say don't make the switch. If you're horsepower happy, then the SRT-6 would be the way to go.