The house was built between 1855/56 by William Glover Joy, seed crusher and oil merchant, on land which earlier had belonged to the Earl of Cardigan's estate.
David Brudenell-Bruce, Earl of Cardigan,Note 2 the current and thirty-first warden of Savernake Forest has initiated the renovation of Tottenham House and a golf course is planned for the southern part of Savernake estate.
On 25 February 1628 Sir Thomas Brudenell (whose family had married into the Wyvilles) was created Baron Brudenell of Stonton.
James Earl Jones | Prince Edward, Earl of Wessex | Louis Mountbatten, 1st Earl Mountbatten of Burma | Herbert Kitchener, 1st Earl Kitchener | Earl | Frederick Roberts, 1st Earl Roberts | Earl of Derby | Earl Warren | Earl of Pembroke | Robert Harley, 1st Earl of Oxford and Earl Mortimer | Earl of Warwick | Thomas Wentworth, 1st Earl of Strafford | Edward Smith-Stanley, 14th Earl of Derby | Earl of Shrewsbury | William Pitt, 1st Earl of Chatham | Simon de Montfort, 6th Earl of Leicester | Richard Neville, 16th Earl of Warwick | Earl of Leicester | John Scott, 1st Earl of Eldon | Robert Devereux, 2nd Earl of Essex | Robert, 1st Earl of Gloucester | Edward Harley, 2nd Earl of Oxford and Earl Mortimer | Earl of Devon | Douglas Haig, 1st Earl Haig | My Name Is Earl | Edward Hyde, 1st Earl of Clarendon | Earl Scruggs | Earl of March | Richard Howe, 1st Earl Howe | John Russell, 1st Earl Russell |
His brother-in-law, the Earl of Cardigan, was one of his subordinates, commanding the Light Brigade – an unfortunate choice as the two men heartily detested each other.
In 1868 the 2nd Marquess also inherited the Earldom of Cardigan from his kinsman the 7th Earl of Cardigan, and so the Marquesses of Ailesbury now also hold the titles Earl of Cardigan (1661) and Baron Brudenell, of Stonton in the County of Leicester (1628), in the Peerage of England, as well as being Baronets of England, styled "of Deene in the County of Northampton".