Yes, there's at least one other forum regular from IoM (sorry, I've forgotten his ID) - maybe that's your neighbour .
I'm not knocking the Z4 at all - it is one of the 'ultimate driving machines' after all and looks great - I've had many Beemers over the years (but not a Z4 - yet!) - but my point is, most people go for a XF at least initially for its very distinctive looks - and I agree with you, the coupe's back end and roof line are what really makes it stand out and, nice though it is, something of that is lost with the 'vert. - that's why I have a coupe (plus the Mrs isn't keen on convertibles - something about her hair, I think...!)
The manual XF does seem to divide opinion - (I think initial press reactions didn't help but they all want perfection and some were biased against it from the very beginning because of its badge) but there are those who swear by them, even if they acknowledge slight shortcomings in gear ratios and fuel consumption - but over all they still seem to have good following.
I guess the only way to be sure is to try one - which I admit I haven't - after looking at a couple of standard autos (too expensive and too tatty), I got lucky and went straight into an SRT6 and haven't looked back - even thought I too usually prefer a manual.

We've only been to IoM once, fantastic place and your super pics have really inspired me to come back and experience those wonderful roads in a XF!