Good idea on not eating at Nepethe, how about lunch at Madonna Inn in San Luis Obispo. It's large, easy to get to, off 101 just before the turn off on 1 to Morro Bay.
This is kinda of self serving, I'm planning on driving down to San Luis Obispo and hooking up with the group when they come through.