I was in your same position a few months ago and just bought a limited.
I have driven several base and limited models.
I have had a manual for 10 years and was ready for an automatic so that was a requirement. The cloth seats are more comfortable IMO. I wanted heated seats but the one I bought has an issue with the seat bottoms not working. Radio... I dont care, NAV, I dont care. The only base models I found local were Black and I wanted Red. I looked at several SE models online and had they been closer, I would have probably bought one.
I would have been very happy with an SE so I say go for it.