Born in New York City, Elmslie, a grandson of publisher Joseph Pulitzer, spent his childhood in Colorado Springs, Colorado, prepped at the St. Mark's School in Southborough, Massachusetts, and graduated from Harvard in 1950 with a B.A. in literature.
In 2000 he revived the Adventures in Poetry imprint with Cris Mattison, publishing books by John Ashbery, John Godfrey, Kit Robinson, Alan Bernheimer, Jean Day, Carla Harryman, North, Coolidge, Kenward Elmslie, and others.