Robert died childless and so the castle passed to his brother Urse d'Abetot's daughters, one of whom, Matilida, married Robert de Marmion.
Little Witley manor passed to the Beauchamp family when Emeline de Abitot, the daughter and heiress of Urse d'Abetot, married Walter de Beauchamp then owner of Elmley Castle.