Born into the recusancy on 2 April 1831 at Stonor, England, the ancestral home of the Stonor family, he was the son of Thomas Stonor, 3rd Lord Camoys and Frances (née Towneley).
Towneley's body was buried in an unmarked grave on 31 July either in the church or churchyard of St. Pancras, London.
Alexander Koch de Gooreynd changed the family name prior to the birth of Towneley's younger brother, Sir Peregrine Worsthorne, the journalist.