Alright. So had time yesterday afternoon to go out and take a closer look. Got my flashlight out and a mirror and was able to see another screw right behind the blower motor resistor in a very hard to reach and awkward place. Took me a little while to get it out but the blower motor came down right after.
It didn't look rusty but there was a few old leafs heavily jammed in the blower and all around the resistor. Cleaned it out, cleaned the connections and used some WD-40 to make sure it all runs smooth. Everything works nice and quiet now.

Only thing that concerns me is how did so many leaf get into the filter compartment and subsequently the blower. Obviously the drains on the left and right of the windshield under the hood were stuffed with leafs. I am seriously considering putting a screen on top of the drain like the ones used for windows or similar where water can still run through the drain but leafs can't get in. I wonder what everyone else has done to alleviate this issue?
Granted I am parked under trees quite a bit but I'm sure others have a similar parking situation?