The house was owned at one time by Sir William Hamilton, Ambassador to the Court of Naples, whose (second) wife was Emma, Lady Hamilton, Lord Nelson's paramour.