His father Cornelius Gurlitt was an architect and art historian, his brother Willibald a musicologist, his sister Cornelia a painter and his cousin Wolfgang was an art dealer as well.
The composer Manfred Gurlitt was his stepbrother, and was a cousin of the musicologist Wilibald Gurlitt.