Brymer was born at Bath, the son of John Brymer of Ilsington, Dorchester, Dorset and his wife Eliza Mary Tugwell, only daughter of George Tugwell of Crowe Hall, near Bath.
Robert's main residence was at Rivenhall in Essex but he also held the manors of Lyneford, Hokewold cum Wiltone, Reynham, South Lenn, Middleton, Berton Bynedick, Hoo and Ilsington in Norfolk.