Nowhere else in the United States, other than the White House, have so many prominent political figures lived in one home - two presidents, two governors, a first lady, CIA director and much more.