# Can KL-Divergence ever be greater than 1?

I've been working on building some test statistics based on the KL-Divergence,

$$D_{KL}(p \| q) = \sum_i p(i) \log\left(\frac{p(i)}{q(i)}\right),$$

And I ended up with a value of $1.9$ for my distributions. Note that the distributions have support of $140$K levels, so I don't think plotting out the whole distributions would be reasonable here.

What I'm wondering is, is it possible to have a KL-Divergence of greater than 1? A lot of the interpretations I've seen of KL-Divergence are based on an upper bound of 1. If it can go greater than 1, what is the interpretation of KL-Divergence beyond 1?

Edit: I know it's a bad choice of reference, but the Wikipedia article on KL Divergence suggests that "a Kullback–Leibler divergence of 1 indicates that the two distributions behave in such a different manner that the expectation given the first distribution approaches zero." I had thought it was implied that this meant the KL-Divergence was bounded above by 1, but it's apparent that this is a mistake in the article.

• Can you please state, and provide references/links as available, for the interpretations you've seen of KL-Divergence which are based on an upper bound of 1? – Mark L. Stone Jan 14 '18 at 18:10
• edited to indicate my (poor) reference. – bearAndShark Jan 14 '18 at 18:17

The Kullback-Leibler divergence is unbounded. Indeed, since there is no lower bound on the $q(i)$'s, there is no upper bound on the $p(i)/q(i)$'s. For instance, the Kullback-Leibler divergence between a Normal $N(\mu_1,\sigma_1^2)$ and a Normal $N(\mu_2,\sigma_1^2)$ is $$\frac{1}{2\sigma_1^{2}}(\mu_1-\mu_2)^2$$which is clearly unbounded.

Wikipedia [which has been known to be wrong!] indeed states

"...a Kullback–Leibler divergence of 1 indicates that the two distributions behave in such a different manner that the expectation given the first distribution approaches zero."

which makes no sense (expectation of which function? why 1 and not 2?)

A more satisfactory explanation from the same Wikipedia page is that the Kullback–Leibler

"...can be construed as measuring the expected number of extra bits required to code samples from P using a code optimized for Q rather than the code optimized for P."

• The KL divergence between two normal distributions does not depend on one of the variances? – amoeba Jan 14 '18 at 19:44
• just to add to this - the eqn Xi'an gives is when the second Gaussian a standard normal distribution (variance of 1). Wiki has both forms en.wikipedia.org/wiki/… – Miss Palmer Aug 19 '18 at 10:27