Are you going to put them back when it comes time for emissions inspection? I guess you must live in a state that doesn't require them.
As far as pollution is concerned, it's more about smog and poor local air quality from exhaust impurities.