Henry Fane was the grandson of Francis Fane, a Restoration dramatist, and son of Henry Fane (1669–1726) of Brympton and Anne, sister and coheir of John Scrope of Wormsley.
The house and estate was passed to his grandson John Scrope a baron of the Exchequer and as Scrope died without issue, his estate of Wormsley passed to the descendants of his sister Anne (died 1720), who had married Henry Fane of Brympton.