On his father's side Isaac was a grandson of R. Simhah ben Samuel of Vitry, author of the Maḥzor Vitry; on his mother's side he was a nephew of Rabbeinu Tam, of Rashbam, and of Isaac ben Meir (RiBaM), a great-grandson of Rashi, and a relative of R. Eleazar of Worms.