Mine does the same thing to a certain extent - not as bad as yours, but I HAVE noticed that it takes more of a movement to downshift than to upshift.

I don't know, but suspect that the switch module/assembly that controls shifting is not mounted exactly right in many or most cars.

Or......... maybe they intended this because they wanted it to be easier to upshift and/or didn't want you to downshift unless you MEANT it.

Oh, and Boiler: I must say, this is the first car I'v owned with an automatic. It is THAT much fun - that I'd buy an XF with an auto!!!!
I tried two 6 speeds, but didn't like how they shifted. I'd still have bought one had I found a 6 speed Roadster - but when this car came along, (low miles, red, excellant condition, etc.) it just had too much going for it to pass up.