Do the windows always go down when you unlatch the top BEFORE you push the down switch ?
If so, then I'd suspect the switch which is located just above the passenger's right ear. It is actuated by a sliding plastic piece.
Could be the switch, wiring or sliding actuator.

If not, then it could be the switch in the header that senses the "D" handle latches being engaged.
( NOTE: this switch is in the windshield header, not in the top itself )