Foveaux was baptised on 6 April 1767 at Ampthill, Bedfordshire, England, the sixth child of Joseph Foveaux and his wife Elizabeth, née Wheeler.
On 18 March 1664, Robert Bruce, 2nd Earl of Elgin in the Peerage of Scotland was created Baron Bruce, of Skelton in the County of York, Viscount Bruce, of Ampthill in the County of Bedford, and Earl of Ailesbury, in the County of Buckingham, all in the Peerage of England.
Ampthill Park was the burial place for the golden hare in the Kit Williams treasure hunt Masquerade.
He profited greatly from the ransoms of prisoners, and used the spoils to build Ampthill Castle, and to buy Oyster Hill Manor from the Mayor of London, William Walworth.