Francis Elliot was born Frances Vickriss Dickinson at Farley Hill Court in the Berkshire village of Swallowfield on 6 March 1820, the only child of Charles Dickinson of Queen Charlton Manor, Somerset.
In 1717, Thomas 'Diamond' Pitt, the Governor of Fort St. George, bought Swallowfield Park from Edward Hyde, reputedly using part of the proceeds of his sale of the Regent Diamond to Philippe II, Duke of Orléans.