I used to be 5 minutes walk away - I prefer being further away, because I don't want to be too close to the students, and I don't want to bump into other faculty outside of times I'm on campus. I also find the some physical separation helps the psychological separation.
You make a good point, but I am really enjoying this, my first semester within walking distance of where I work. I eat lunch at home every day, with a few rare exceptions. I'm pregnant and really appreciate being so close to home to be able to put my feet up or lie down for 45 minutes or an hour. If I had a longer commute, even 10 or 15 minutes one way, which isn't long, I wouldn't be able to do this as often.