Baptised Dudley Long at Saxmundham, Suffolk, he was the younger of two sons of Charles Long (1705–1778), landowner, of Hurts Hall, Suffolk, and his wife, Mary, daughter and coheir of Dudley North of Little Glemham, Suffolk, and granddaughter of Sir Dudley North.
Little Glemham has a church called St Andrew Church, a pub and a hall called Glemham Hall.