I had a 2005 Saab Aero convertible before the XF and the noise was comparable with the top up (the Saab had an interior lining for the top which the XF does not).

I believe that the reasons there is so much noise with the top are as follows:

1) gaps in weather strip allow direct entry of sound - sounds coming through cracks are crisp, as if the top was down. You can usually find the source of these by listening.
2) sound waves hit the cloth top, deflecting it (just like a speaker cone) which results in compression/expansion of the air in the cockpit of the car. These sounds are muffled.

I am not sure there is anything you can do. Even if you fill all the gaps, you will still have the muffled sounds due to the lack of rigidity of the convertible material.