Yoritomo was the third son of Minamoto no Yoshitomo, heir of the Minamoto (Seiwa Genji) clan, and his official wife, a daughter of Fujiwara no Suenori, who was a member of the illustrious Fujiwara clan.