Stop you don't need to pull the door panel to get the handle off.

Pull back the weather stripping on the door and you will find a hole that lines up with the hex screw that holds the handle on. If I remember its 5mm. remove screw and make sure you use a magnet on the hex key to pull it through the door. This will undo the right most part of the handle and then the large portion tilts out