Battery is easiest to replace, how old is it ?
The voltage regulator is in the alternator, you can kill 2 birds with one stone there.

Check that you have a Valeo or Bosch alternator before ordering.