Followup:

ended up taking the XF to the shop since I couldn't jack the car up high enough to get good enough access to the starter (specifically it wasn't high enough to take the parts of the exhaust off to get to it). Turns out it was the starter.

It's good as new, now :-)