It might be the road "ruts". Since it isn't doing it on the freeway.

Find some road where it is pulling and see if the car always pulls the same way on that particular piece of road.

.02