When I had my manual tranny I had this problem and I think it was because I had not fully put it in gear. So I made it a habit to do it positively.
The original 04s had a TSB out on this very problem.
Going to second and then into first seemed to help.
First gear is pretty useless anyway, if I was on a slight downhill I would start in second.