Born in Ireland to a Hugenot family, he was the son of William Richard Le Fanu (1816- 1894) and his wife Henrietta Victorine Barrington, daughter of Sir Matthew Barrington, 2nd Baronet.