The king of Delhi, Iltutmish, is known to have attacked and sacked Vidisha in the thirteenth century and this would have given him an opportunity to remove the pillar as a trophy to Delhi, just as the Tughluq rulers brought Asokan pillars to Delhi in the 1300s.
Muhammad Shah ibn Feroze Shah after a brief battle took shelter in the fort of Nagarkot, and Tughluq Khan’s army returned to Delhi without pursuing him any further due to the difficulties of the venture & terrain.