The house was built in 1632 for the Herbert family, and its first known occupant was William Herbert (brother of Richard Herbert, Lord of Cherbury), who inherited it from their father, Edward Herbert.