Among Gem's sporting interests was the game of rackets, which he played at the Bath Street Racquets Club adjacent to the Racquet Court Inn in Bath Street, Birmingham with his friend Augurio Perera, a Spanish merchant based in Birmingham.
Squash, for the 2013 Island Games, took place at the Bermuda Squash Racquets Association in Devonshire Parish, Bermuda.
The games of squash racquets and its parent sport, racquets, spread to America in the 1880s with the nation's first courts built at St. Paul's School, in Concord, New Hampshire.