I went to UCSB for my freshman year of undergrad and couldn't get out of there fast enough. It's a good school, but the undergraduate culture was so mind-numbingly party oriented that I couldn't find my niche. This was a while ago, so things may have changed. I doubt it, though.
I doubt that it's changed. I just don't think it's all that different from many institutions. I know ppl that went to Berkeley and UCLA and left to go UCSB to start anew for much the same reasons that you left UCSB. And I know countless ppl from countless schools who partied way too hard their freshman year and either were forced to transfer or got their act together.
But for a professor considering UCSB, it's not that big of a deal. There *is* a gap btw UCB/UCLA and UCSB as far as the undergrads and grads. However, part of the joy of working at these schools is that the faculty becomes really adept at finding young stars in the making and then nurturing them along, moreso than at the elite-tier schools in state systems.
As a prof at UCSB, the OP will have the resources to conduct cutting edge research. So if it's time to move on, that's entirely possible.
OP has a kid, right? SB is home to numerous festivals and almost all of them incorporate children's activities in an incredible way. I was passing through once and there was a chalk festival in which artists painted works of art in chalk on the sidewalk near the mission. And they had workshops for kids and places for the kids to draw.
The kids were thoroughly engaged and having a blast! I made a comment to a few locals and they said that all the festivals do stuff like that.
And then there's Fiestas in August. Girls spend the entire year rehearsing flamenco and put on performances. Stunning!
It's a great place to raise a child, I think.