Williams was born in 1856 in Llowes, Radnorshire, the second child of the Reverend Garnons Williams of Abercamlais, Powys, and his wife Catherine Frances, the daughter of Fenton Hort, of Leopardstown, Dublin, and sister of Fenton John Anthony Hort.