Just an update on my rewiring. It's been a few weeks since I bypassed the internal amp on my head unit and wired the RCA pre-outs directly to the stock amp and I couldn't be happier. For some reason before I made the change, I could just never get the sound the way I wanted. There was always something a bit...off... and I could never really figure it out. Well now the sound is great! Less noise, more definition, less bass distortion. There is less volume but it's still plenty loud enough when I get above 30 (the max is 35 on my HU).

Thanks to everybody for suggesting this. I'm really picky about how my music sounds and I'm finally able to just listen without messing with the internet EQ all the time now.