Skip to main content
Cross Validated

Return to Question

Tweeted twitter.com/StackStats/status/952711272797294592
edited tags
Link
Xi'an
Xi'an
  • 107.7k
  • 13
  • 190
  • 676
added references to my interpretation
Source Link
plbmr
plbmr
  • 123
  • 1
  • 1
  • 5

I've been working on building some test statistics based on the KL-Divergence,

\begin{equation} D_{KL}(p \| q) = \sum_i p(i) \log\left(\frac{p(i)}{q(i)}\right), \end{equation}

And I ended up with a value of $1.9$ for my distributions. Note that the distributions have support of $140$K levels, so I don't think plotting out the whole distributions would be reasonable here.

What I'm wondering is, is it possible to have a KL-Divergence of greater than 1? A lot of the interpretations I've seen of KL-Divergence are based on an upper bound of 1. If it can go greater than 1, what is the interpretation of KL-Divergence beyond 1?

Edit: I know it's a bad choice of reference, but the Wikipedia article on KL Divergence suggests that "a Kullback–Leibler divergence of 1 indicates that the two distributions behave in such a different manner that the expectation given the first distribution approaches zero." I had thought it was implied that this meant the KL-Divergence was bounded above by 1, but it's apparent that this is a mistake in the article.

I've been working on building some test statistics based on the KL-Divergence,

\begin{equation} D_{KL}(p \| q) = \sum_i p(i) \log\left(\frac{p(i)}{q(i)}\right), \end{equation}

And I ended up with a value of $1.9$ for my distributions. Note that the distributions have support of $140$K levels, so I don't think plotting out the whole distributions would be reasonable here.

What I'm wondering is, is it possible to have a KL-Divergence of greater than 1? A lot of the interpretations I've seen of KL-Divergence are based on an upper bound of 1. If it can go greater than 1, what is the interpretation of KL-Divergence beyond 1?

I've been working on building some test statistics based on the KL-Divergence,

\begin{equation} D_{KL}(p \| q) = \sum_i p(i) \log\left(\frac{p(i)}{q(i)}\right), \end{equation}

And I ended up with a value of $1.9$ for my distributions. Note that the distributions have support of $140$K levels, so I don't think plotting out the whole distributions would be reasonable here.

What I'm wondering is, is it possible to have a KL-Divergence of greater than 1? A lot of the interpretations I've seen of KL-Divergence are based on an upper bound of 1. If it can go greater than 1, what is the interpretation of KL-Divergence beyond 1?

Edit: I know it's a bad choice of reference, but the Wikipedia article on KL Divergence suggests that "a Kullback–Leibler divergence of 1 indicates that the two distributions behave in such a different manner that the expectation given the first distribution approaches zero." I had thought it was implied that this meant the KL-Divergence was bounded above by 1, but it's apparent that this is a mistake in the article.

Source Link
plbmr
plbmr
  • 123
  • 1
  • 1
  • 5

Can KL-Divergence ever be greater than 1?

I've been working on building some test statistics based on the KL-Divergence,

\begin{equation} D_{KL}(p \| q) = \sum_i p(i) \log\left(\frac{p(i)}{q(i)}\right), \end{equation}

And I ended up with a value of $1.9$ for my distributions. Note that the distributions have support of $140$K levels, so I don't think plotting out the whole distributions would be reasonable here.

What I'm wondering is, is it possible to have a KL-Divergence of greater than 1? A lot of the interpretations I've seen of KL-Divergence are based on an upper bound of 1. If it can go greater than 1, what is the interpretation of KL-Divergence beyond 1?