When you apply a Carnauba wax to your car's clearcoat, and you rub it with anything - your fingers, sponge, pads, ANYTHING, it is basically REMOVING your clearcoat a little bit at a time. This is how scratches are removed by waxing.
I once owned a '79 Corvette with a swooped up 350 L82 engine. (I sold it and bought a Hyundai when i got married...Ouch).
I had a built in spoiler and custom paint. The shop applied 7 coats of clear-coat giving the paint depth and resiliency. No amount of polish deterred that mirror finish...
I'd have to agree that waxes contain abrassives however what is the alternative?
I can assure you I'd never again spend the money on a custom paint job.
I think the only alternative is to Wax sparingly with a high quality wax - Twice/year tops.
If you don't wax I fear the elements, dirt and grime would exact a far greater toll on the paint than the mild abrasive wear caused by waxing. That smooth surface repells dust, which BTW is also an abrassive.

My black Altima still has a great Non swirl finish after 4 years...I don't let anyone wash her aside from touchless, but me.