What's an intuitive way to understand how KL divergence differs from other similarity metrics?

Question

The general intuition I have seen for KL divergence is that it computes the difference in expected length sampling from distribution $P$ with an optimal code for $P$ versus sampling from distribution $P$ with an optimal code for $Q$.

This makes sense as a general intuition as to why it's a similarity metric between two distributions, but there are a number of similarity metrics between two distributions. There must be some underlying assumptions based on how it chooses to assign distance versus other metrics.

This seems fundamental to understanding when to use KL divergence. Is there a good intuition for understanding how KL divergence differs from other similarity metrics?

Answer 1

A very short answer; there are too many similarity metrics (or divergences) proposed to even try looking at more than a few. I will try to say a little about why use specific ones.

• Kullback-Leibler divergence: See Intuition on the Kullback-Leibler (KL) Divergence, I will not rewrite here. Short summary, KL divergence is natural when interest is in hypothesis testing, as it is the expected value under the alternative hypothesis of the log likelihood ratio. Some other divergences look at other functions of the likelihood ratio, but log is natural given its role in statistical inference.

• Earth Mover distance, see Difference between Hausdorff and earth mover (EMD) distance and Wikipedia. The ideas here are very different from KL divergence, and I cannot see obvious connection to inference. The wikipedia article gives the following example:

An early application of the EMD in computer science was to compare two grayscale images that may differ due to dithering, blurring, or local deformations.[10] In this case, the region is the image's domain, and the total amount of light (or ink) is the "dirt" to be rearranged.

This seems similar to dynamic time warping used in time series.

• Bhattacharyaa distance, see Intuition of the Bhattacharya Coefficient and the Bhattacharya distance?. This is also related to inference, it is the expectation under the null hypothesis of the square root of the likelihood ratio. To me it is unclear why it is interesting, but it can be seen as a generalization of the Mahalanobis distance to nonnormal distributions. Note that starting from $\int \left(\sqrt{f(x)} - \sqrt{g(x)}\right)^2 \; dx \geq 0$ a little manipulation gives $\int \sqrt{f(x) g(x) }\; dx \le 1$ for densities $f, g$. That might give some intuition.

• Chisquare distance can be found here have a lot of tradition and seems natural with discrete data. An example of use is correspondence analysis.

Probably many divergences are mostly used technically in proofs, and intuition must then come from their use. An interesting paper.