# Can KL-Divergence ever be greater than 1?

I've been working on building some test statistics based on the KL-Divergence,

\begin{equation} D_{KL}(p \| q) = \sum_i p(i) \log\left(\frac{p(i)}{q(i)}\right), \end{equation}

And I ended up with a value of $1.9$ for my distributions. Note that the distributions have support of $140$K levels, so I don't think plotting out the whole distributions would be reasonable here.

What I'm wondering is, is it possible to have a KL-Divergence of greater than 1? A lot of the interpretations I've seen of KL-Divergence are based on an upper bound of 1. If it can go greater than 1, what is the interpretation of KL-Divergence beyond 1?

Edit: I know it's a bad choice of reference, but the Wikipedia article on KL Divergence suggests that "a Kullback–Leibler divergence of 1 indicates that the two distributions behave in such a different manner that the expectation given the first distribution approaches zero." I had thought it was implied that this meant the KL-Divergence was bounded above by 1, but it's apparent that this is a mistake in the article.

• Can you please state, and provide references/links as available, for the interpretations you've seen of KL-Divergence which are based on an upper bound of 1? Jan 14, 2018 at 18:10
• edited to indicate my (poor) reference. Jan 14, 2018 at 18:17

The Kullback-Leibler divergence is unbounded. Indeed, since there is no lower bound on the $$q(i)$$'s, there is no upper bound on the $$p(i)/q(i)$$'s. For instance, the Kullback-Leibler divergence between a Normal $$N(\mu_1,\sigma_1^2)$$ and a Normal $$N(\mu_2,\sigma_1^2)$$ is $$\frac{1}{2\sigma_1^{2}}(\mu_1-\mu_2)^2$$which is clearly unbounded.

Wikipedia [which has been known to be wrong!] indeed states

"...a Kullback–Leibler divergence of 1 indicates that the two distributions behave in such a different manner that the expectation given the first distribution approaches zero."

which makes no sense (expectation of which function? why 1 and not 2?)

A more satisfactory explanation from the same Wikipedia page is that the Kullback–Leibler divergence

"...can be construed as measuring the expected number of extra bits required to code samples from P using a code optimized for Q rather than the code optimized for P."

• The KL divergence between two normal distributions does not depend on one of the variances? Jan 14, 2018 at 19:44
• just to add to this - the eqn Xi'an gives is when the second Gaussian a standard normal distribution (variance of 1). Wiki has both forms en.wikipedia.org/wiki/… Aug 19, 2018 at 10:27
• What exactly do you mean with unbounded? The KL-Divergence bounded from below because it cannot become negative? Sep 12, 2020 at 7:27
• Unbounded: having no upper bound (since the lower bound is zero). Sep 12, 2020 at 7:51