What should we be prying against?

What should we use as the fulcrum?

We didn't want to break anything and cause further damage, so we went a bit easy on it when we used a large screw driver to attempt to push it back.