Welcome to the forum.
Personally, I think your fine with any of the 5 years of production. I would simply try to find as new, and as low mileage as possible for the money you have to spend.
Some early 04's had problems with the transmission popping out of gear, but I would think most any one you would find now-a-days had the problem taken care of by now.
I think the valve cover gasket leak is pretty universal for all the years. My Daughter had her's fixed for under $200.
I purchased my 04 6spd. brand new on April 1st. of 05. With over 70k miles, the only warranty work I've had was a new battery 3 days before the 3 year 36k mile warranty.
Being a previous "Conquest" owner (those cars are so cool) you may be a little disappointed with the Crossfires "notchy" 6spd. Actually, the 1st. to 2nd. shift is the worst of any sports car I've owned, and I've had a few. That being said, I still love having a 6spd. and it seems some shift better than others. I think it's just something you get use to.
Take your time shopping, and don't hesitate to ask for help here on the forum. It's the greatest group of "Car People" you will ever want to meet. We all love our cars.