After doing this a few times, I found that I could remove the carpeted side pieces. Then pry forward on the black rear panel and slip both foam molded pieces out one at a time. I takes 2 to get it done but it works. If you heat a blade and trim the tops of the foam molded inserts it makes it much easier the next time.

The only way I have found to get the rear most black panel out is to remove the left and right upper cream colored panels.