IMHO...
The problem is that the Crossfire doesn't deliver on it's looks. Top Gear loved the look of the thing but pointed out that it did have some flaws:
1) Crappy plastic interior painted silver (look at the competition and you'll see real metal)
2) A 3.2L engine should be more powerful, if the S2000 can do 240 bhp from 2L, why do we get less with over 50% more displacement
3) Poor steering and feedback - not bad, it does the job, but when you're caning it round zig zag country roads, you want as much feedback as possible. Drive even just an MX5/Miata and you'll discover what the Crossfire is lacking.
4) Poor gears - Mercedes are famed for their poor manual gearbox, and didn't prove anybody wrong. European driving is stop/start, quick bursts up to 60 mph, then stomping on the breaks when you see a speed trap, the gears are OK, but no where as slick as rivals. Oh and in the UK, automatics are for old people ;-)
5) Price - and this is the killer. In the UK this was originally on sale for £28,000.. that's USD 52,000, heading up to Porsche Boxter prices and directly competing with the S2000 and 350Z.

I bought my Crossfire after driving and S2000 for five years, but I got it for £4000 below list price and bought it knowing that it wouldn't be as involving a drive. I was older (not necessarily wiser) and wanted a more civilized ride and something that's rare and beautiful (much like the S2000 five years ago).

Top Gear was right to criticize it... Chrysler was greedy, and priced it into the wrong bracket, they don't have the strong brand in Europe that will allow them to charge top dollar, so the Crossfire needed to be both better than all of the direct competition and even the sector above it to get the sales. If it would have been priced at £20K, it would have been hailed as the best car for your money and Chrysler would be beating the customers off with a stick, and the whole brand would have been strengthened.