I got it done. For those that may come across this in the future it really is just a matter of carefully prying the speaker away. I used a small Flathead screwdriver, started at the opening where the cord goes and worked my way around.