I left my first TT position when I was asked (in a department meeting) what makes YOU think you're qualified to teach (junior-level core course).
I left my second TT position due primarily to politics (liberal closeted gay male surrounded by Rush Limbaugh-listening conservatives, in a small town).
I am now tenured at my third institution, which is a wonderful fit. There is hope, though I know I'm extremely fortunate to have ended up on my feet.
Good for you! Yes, it can work out, but it is always a risk. Of course, nothing ventured, nothing gained. I left my first TT job because of family, took a visiting job and it turned into a tt and now I am on the other side. As you say, I too feel extremely fortunate!