Sensitivity of KL Divergence

Question

I am very new to the concept of KL divergence. Although I have grasped the fundamental formulations, I have a confusion comparing the KL divergence across the different distributions. Suppose I have 3 random distributions [$y_1$,$y_2$ and $y_3$] and I want to compare with a known distribution $\textbf{x}$. How do I determine the best distribution that matches the distribution of $\textbf{x}$? I understand the lower the KL divergence, the better matching, but what would be the sensitivity of the KL divergence parameter, following which we can say that the distribution is significantly different?

For example, what would be the method to get the threshold upon which I can say that all distributions below this threshold are similar?

Answer 1

The question

How do I determine the best distribution that matches the distribution of $\textbf{x}$?"

is much more general than the scope of the KL-divergence (also known as relative entropy). And if a goodness-of-fit like result is desired, it might be better to first take a look at tests such as the Kolmogorov-Smirnov, Shapiro-Wilk, or Cramer-von-Mises test. I believe those tests are much more common for questions of goodness-of-fit than anything involving the KL-divergence.

The KL-divergence is more commonly used as a measure of information gain, when going from a prior distribution to a posterior distribution in Monte Carlo simulations.

---

All that said, here we go with my actual answer:

Note that the Kullback-Leibler divergence from $q$ to $p$, defined through $$ D_{KL}(p|q) = \int p \log\left(\frac{p}{q}\right) dx $$ is not a distance, since it is not symmetric and does not meet the triangular inequality. It does satisfy positivity $D_{KL}(p|q)\ge0$, though, with equality holding if and only if $p=q$. As such, it can be viewed as a measure of discrepancy and can indeed be used in goodness-of-fit tests.

This is used e.g. in R's package vsgoftest implementing the Vasicek-Song test, which will provide you with a p-value where you can use the commonly used threshold of 0.05. For more details on this, have a look at the package author's paper arXiv:1806.07244 or at one of the original papers:

• Vasicek, O., A test for normality based on sample entropy, Journal of the Royal Statistical Society, 38(1), 54-59 (1976).
• Song, K. S., Goodness-of-fit tests based on Kullback-Leibler discrimination information, Information Theory, IEEE Transactions on, 48(5), 1103-1117 (2002).