I've done it with both the engine running and not, but usually with it running. As James said, having the motor running will give you a bit more voltage to work with and may put the top up or down slightly faster (probably less than a one second difference).