Born at Toddington in Gloucestershire, he was fifth son of Robert Tracy, 2nd Viscount Tracy of Rathcoole, he was the eldest son of his second wife, Dorothy, daughter of Thomas Cocks of Castleditch, Herefordshire.