Removing pulleys are often very easy or very difficult.
You need to jiggle it both up down side to side while pulling it towards you. If you have pulled too much toward you try pushing it in again up down or side to side. Pulling it off should be done in an angle aswell, after you pulled it out approx 1 cm straight out.
Hard to describe but jiggle it is my best bet. I was lean aswell.