The tiny flecks are essentially impact damage from tiny bits of stuff hitting the car at speed. They are a sign that the car is driven on public roads - yes, it happens to all of us, it's just more noticeable on certain colors, primarily non metallic paints. There really isn't much you can do about them, but you certainly did not cause them by doing what you did. All the swirls that were present before you started actually masked these little pits, which is why you didn't notice them before.

Overall though, you did an outstanding job removing swirls from what is, in my experience, very hard paint. Good for you!!