What's an intuitive way to understand how KL divergence differs from other similarity metrics?

The general intuition I have seen for KL divergence is that it computes the difference in expected length sampling from distribution $$P$$ with an optimal code for $$P$$ versus sampling from distribution $$P$$ with an optimal code for $$Q$$.

This makes sense as a general intuition as to why it's a similarity metric between two distributions, but there are a number of similarity metrics between two distributions. There must be some underlying assumptions based on how it chooses to assign distance versus other metrics.

This seems fundamental to understanding when to use KL divergence. Is there a good intuition for understanding how KL divergence differs from other similarity metrics?

• By bias, do you mean why people seem to like KL divergence (have a bias for KL over other distribution difference measures) or a bias in the statistical sense of the expected value of a parameter estimate not equaling the parameter? (I assume the former, but bias is a technical term in statistics.)
– Dave
Aug 14 '20 at 0:58
• Ah great distinction I mean the former. I'm really asking if you need a distance metric, what features of the problem should lead you to use KL divergence instead of another metric. Aug 14 '20 at 1:07
• Given your explanation in comments, I have reframed your question to remove reference to 'bias' and more accurately capture your interest. Please check the new wording to see if it correctly captures what you want to know.
– Ben
Aug 16 '20 at 22:22
• Does this answer your question? Intuition on the Kullback-Leibler (KL) Divergence Sep 24 '20 at 16:32

A very short answer; there are too many similarity metrics (or divergences) proposed to even try looking at more than a few. I will try to say a little about why use specific ones.

• Kullback-Leibler divergence: See Intuition on the Kullback-Leibler (KL) Divergence, I will not rewrite here. Short summary, KL divergence is natural when interest is in hypothesis testing, as it is the expected value under the alternative hypothesis of the log likelihood ratio. Some other divergences look at other functions of the likelihood ratio, but log is natural given its role in statistical inference.

• Earth Mover distance, see Difference between Hausdorff and earth mover (EMD) distance and Wikipedia. The ideas here are very different from KL divergence, and I cannot see obvious connection to inference. The wikipedia article gives the following example:

An early application of the EMD in computer science was to compare two grayscale images that may differ due to dithering, blurring, or local deformations.[10] In this case, the region is the image's domain, and the total amount of light (or ink) is the "dirt" to be rearranged.

This seems similar to dynamic time warping used in time series.

• Bhattacharyaa distance, see Intuition of the Bhattacharya Coefficient and the Bhattacharya distance?. This is also related to inference, it is the expectation under the null hypothesis of the square root of the likelihood ratio. To me it is unclear why it is interesting, but it can be seen as a generalization of the Mahalanobis distance to nonnormal distributions. Note that starting from $$\int \left(\sqrt{f(x)} - \sqrt{g(x)}\right)^2 \; dx \geq 0$$ a little manipulation gives $$\int \sqrt{f(x) g(x) }\; dx \le 1$$ for densities $$f, g$$. That might give some intuition.

• Chisquare distance can be found here have a lot of tradition and seems natural with discrete data. An example of use is correspondence analysis.

Probably many divergences are mostly used technically in proofs, and intuition must then come from their use. An interesting paper.