I too have noticed this with my 2004 6 speed manual XF. I only notice the noise when i put the car into neutral and the engine is idling and just as you said the noise goes away when i put the clutch in.

I have since changed out the original spark plugs and fuel filter (last friday) and have not noticed the strange noise since. My guess is that one of those components was causing the engine to idle to low and the clutch didn't like it.