Question:

How much do you have to build the latch up with tape/whatever?

A longer term fix might be to use "spiral wrap" instead. No disassembly and about .020" build up. It's really cheap and can be found at The Home Depot in the electrical dept.