Although it's an issue now, it won't be too long before the relatively few years between the first & last XFs will be less relevant to value. Then, rightly, age mileage & general condition will be the overriding factors.
I recently sold my 1973 BMW 2002tii (to raise funds for my SRT) - they were also made for a similar 4-year period (71-75) and, apart from facelift detail changes, anyone wanting a good tii will not care what year it is - and it was that way over 20 years ago when I bought my first one. OK, some of us might not see our XFs' 40th birthdays but you get the idea!

As things are now though, I'd start by thinking of it as a very nice standard colour '04 at £4k - then add £500+ for the very low mileage, then the (depreciated?) value of the extras, depending on how important they are to a prospective buyer - everyone's priorities are different.
I've noticed some sellers suggesting they could remove some mods if necessary.
Wish it was a manual with cedar interior, then I might be raiding the piggy-bank...