5
$\begingroup$

I've been working on building some test statistics based on the KL-Divergence,

\begin{equation} D_{KL}(p \| q) = \sum_i p(i) \log\left(\frac{p(i)}{q(i)}\right), \end{equation}

And I ended up with a value of $1.9$ for my distributions. Note that the distributions have support of $140$K levels, so I don't think plotting out the whole distributions would be reasonable here.

What I'm wondering is, is it possible to have a KL-Divergence of greater than 1? A lot of the interpretations I've seen of KL-Divergence are based on an upper bound of 1. If it can go greater than 1, what is the interpretation of KL-Divergence beyond 1?

Edit: I know it's a bad choice of reference, but the Wikipedia article on KL Divergence suggests that "a Kullback–Leibler divergence of 1 indicates that the two distributions behave in such a different manner that the expectation given the first distribution approaches zero." I had thought it was implied that this meant the KL-Divergence was bounded above by 1, but it's apparent that this is a mistake in the article.

$\endgroup$
  • 3
    $\begingroup$ Can you please state, and provide references/links as available, for the interpretations you've seen of KL-Divergence which are based on an upper bound of 1? $\endgroup$ – Mark L. Stone Jan 14 '18 at 18:10
  • $\begingroup$ edited to indicate my (poor) reference. $\endgroup$ – bearAndShark Jan 14 '18 at 18:17
15
$\begingroup$

The Kullback-Leibler divergence is unbounded. Indeed, since there is no lower bound on the $q(i)$'s, there is no upper bound on the $p(i)/q(i)$'s. For instance, the Kullback-Leibler divergence between a Normal $N(\mu_1,\sigma_1^2)$ and a Normal $N(\mu_2,\sigma_1^2)$ is $$\frac{1}{2\sigma_1^{2}}(\mu_1-\mu_2)^2$$which is clearly unbounded.

Wikipedia [which has been known to be wrong!] indeed states

"...a Kullback–Leibler divergence of 1 indicates that the two distributions behave in such a different manner that the expectation given the first distribution approaches zero."

which makes no sense (expectation of which function? why 1 and not 2?)

A more satisfactory explanation from the same Wikipedia page is that the Kullback–Leibler

"...can be construed as measuring the expected number of extra bits required to code samples from P using a code optimized for Q rather than the code optimized for P."

$\endgroup$
  • 1
    $\begingroup$ The KL divergence between two normal distributions does not depend on one of the variances? $\endgroup$ – amoeba Jan 14 '18 at 19:44
  • $\begingroup$ just to add to this - the eqn Xi'an gives is when the second Gaussian a standard normal distribution (variance of 1). Wiki has both forms en.wikipedia.org/wiki/… $\endgroup$ – Miss Palmer Aug 19 '18 at 10:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.