It's not terribly difficult. The top part doesn't need to be removed to swap interiors (I just did mine from all gray to the 2 tone gray)...

Basically you do have to remove the whole interior though. Seats, center/rear console, under the steering wheel part, glove box. If you go slow and save/mark all the screws you should be fine. Just know that if it doesn't come off with ease, there is probably a screw you are missing and if you keep tugging you will break something.

While swapping make sure to do the sticky key fix!