The general intuition I have seen for KL divergence is that it computes the difference in expected length sampling from distribution $P$ with an optimal code for $P$ versus sampling from distribution $P$ with an optimal code for $Q$.

This makes sense as a general intuition as to why it's a similarity metric between two distributions, but there are a number of similarity metrics between two distributions. There must be some underlying assumptions based on how it chooses to assign distance versus other metrics.

This seems fundamental to understanding when to use KL divergence. Is there a good intuition for understanding how KL divergence differs from other similarity metrics?

  • $\begingroup$ By bias, do you mean why people seem to like KL divergence (have a bias for KL over other distribution difference measures) or a bias in the statistical sense of the expected value of a parameter estimate not equaling the parameter? (I assume the former, but bias is a technical term in statistics.) $\endgroup$
    – Dave
    Commented Aug 14, 2020 at 0:58
  • $\begingroup$ Ah great distinction I mean the former. I'm really asking if you need a distance metric, what features of the problem should lead you to use KL divergence instead of another metric. $\endgroup$ Commented Aug 14, 2020 at 1:07
  • 1
    $\begingroup$ Given your explanation in comments, I have reframed your question to remove reference to 'bias' and more accurately capture your interest. Please check the new wording to see if it correctly captures what you want to know. $\endgroup$
    – Ben
    Commented Aug 16, 2020 at 22:22
  • 1
    $\begingroup$ This should help you: stats.stackexchange.com/questions/188903/… $\endgroup$ Commented Aug 16, 2020 at 22:32
  • $\begingroup$ Does this answer your question? Intuition on the Kullback-Leibler (KL) Divergence $\endgroup$ Commented Sep 24, 2020 at 16:32

1 Answer 1


A very short answer; there are too many similarity metrics (or divergences) proposed to even try looking at more than a few. I will try to say a little about why use specific ones.

  • Kullback-Leibler divergence: See Intuition on the Kullback-Leibler (KL) Divergence, I will not rewrite here. Short summary, KL divergence is natural when interest is in hypothesis testing, as it is the expected value under the alternative hypothesis of the log likelihood ratio. Some other divergences look at other functions of the likelihood ratio, but log is natural given its role in statistical inference.

  • Earth Mover distance, see Difference between Hausdorff and earth mover (EMD) distance and Wikipedia. The ideas here are very different from KL divergence, and I cannot see obvious connection to inference. The wikipedia article gives the following example:

An early application of the EMD in computer science was to compare two grayscale images that may differ due to dithering, blurring, or local deformations.[10] In this case, the region is the image's domain, and the total amount of light (or ink) is the "dirt" to be rearranged.

This seems similar to dynamic time warping used in time series.

  • Bhattacharyaa distance, see Intuition of the Bhattacharya Coefficient and the Bhattacharya distance?. This is also related to inference, it is the expectation under the null hypothesis of the square root of the likelihood ratio. To me it is unclear why it is interesting, but it can be seen as a generalization of the Mahalanobis distance to nonnormal distributions. Note that starting from $\int \left(\sqrt{f(x)} - \sqrt{g(x)}\right)^2 \; dx \geq 0$ a little manipulation gives $\int \sqrt{f(x) g(x) }\; dx \le 1$ for densities $f, g$. That might give some intuition.

  • Chisquare distance can be found here have a lot of tradition and seems natural with discrete data. An example of use is correspondence analysis.

Probably many divergences are mostly used technically in proofs, and intuition must then come from their use. An interesting paper.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.