Later the Lords of Fraunberg owned the Castle, then from 1554-1672 it was the property of the Fuggers.