Edinburgh: The castle and Royal Mile, as already suggested. The National Museum of Scotland is also worth a visit.
I would avoid the castle if you are there for the week either side of Easter - it will be mobbed by tourists. In fact, I would avoid the castle during all major vacations: it is a great place to visit, but not with hordes milling around (and yes, since you ask, I do have distinctly asocial tendencies).