In 1841 he married Catherine Duff (c.1820 – 1869), the daughter of General Sir Alexander Duff and sister of James Duff, 5th Earl Fife.
James Bond | James Joyce | James Brown | James Cook | James Stewart | Fife | James II of England | James Garner | James | James Cameron | James Taylor | James Madison | James May | Hilary Duff | Henry James | James Cagney | James II | James Caan | James Earl Jones | Prince Edward, Earl of Wessex | Louis Mountbatten, 1st Earl Mountbatten of Burma | LeBron James | James Monroe | Herbert Kitchener, 1st Earl Kitchener | James Franco | James I | William James | James Wyatt | James, son of Zebedee | James Dean |
He was devoted to his wife Lady Agnes Duff, the youngest daughter of James Duff, 5th Earl Fife by his wife, Lady Agnes Hay, daughter of William Hay, 18th Earl of Erroll and granddaughter of King William IV by his mistress, the actress Mrs. Dorothy Jordan.
The direct line of the chiefs of Clan Skene died out in 1827 and the estates passed to a nephew, James Duff, 4th Earl Fife.
Born at Banff, Scotland, the son of Banff Sheriff Clerk James Duff (1729–1804) by his marriage to Helen Skene 1734–1764, he was a kinsman (first cousin once removed) to the second and third Earls of Fife.
John Lamont a highly regarded astronomer was born at Corriemulzie near Inverey the son of Robert Lamont (forester to James, 2nd Earl Fife) and his wife Elizabeth Ewan.
Thereupon Duff sought distraction in 1808 by volunteering to join the Spaniards in their war against Napoleon.
Rothiemay Castle, partly dating from the 15th century, was rebuilt as a baronial country house in 1788, by James Duff, 2nd Earl Fife.