The only thing that would bother me about a (possible) exhaust leak, even if it was very minor, and stopped on warm (due to expansion), was the possibility of fumes seaping into the car.

Although if this was the case, you should notice if it had entered your interior (with that unique smell).

Couldn't hurt to jack it up, or put on a lift and check around (especially for corrosion at any joints)...