We can define a distance measure between two instances using the Kullback–Leibler (KL) divergence, which calculates the “distance” between two probability distributions.
It was originally introduced by Solomon Kullback and Richard Leibler in 1951 as the directed divergence between two distributions.
Solomon Kullback | point of divergence | Kullback–Leibler divergence | Isi Leibler | Genetic divergence | Bregman divergence |
The inequality in the above form was proved independently by Kullback, Csiszár, and Kemperman.