12
$\begingroup$

I notice in KL-Divergence formula a $ln$ function is used:

$${D_{KL}}(P||Q) = \sum\limits_i {P(i)} \ln \frac{{P(i)}}{{Q(i)}},$$ where $i$ is a point and $P(i)$ the true discrete probability distribution and $Q(i)$ is the approximate distribution. Can anyone help explain why the $ln$ function is used here?

Why it is not simply $${D_{KL}}(P||Q) = \sum\limits_i {P(i)} \frac{{P(i)}}{{Q(i)}}?$$ Is there any special purpose?

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ I truly don't know why someone voted it down. Maybe your are a very sophisticated statistician and think this question is naive,however, for a starter this could be a very important question to help them better understand the some essential principles. $\endgroup$
    – Ralph B.
    Commented Oct 13, 2014 at 12:37
  • 2
    $\begingroup$ I hate to see anonymous downvotes, but I can guess at the motivation behind this one. Because answers can be obtained from basic matter in the Wikipedia article, as well as on our site (such as stats.stackexchange.com/a/6919 which discusses such generalizations of the KL divergence, found with a search on "KL divergence log interpretation"), the downvote might have been applied to indicate a "lack of research." $\endgroup$
    – whuber ♦
    Commented Oct 13, 2014 at 17:17
  • $\begingroup$ See also stats.stackexchange.com/questions/188903/… $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jul 19, 2019 at 7:45

2 Answers 2

Reset to default
7
$\begingroup$

This is somewhat intuitive, I hope I give some ideas.

The KL divergence has several mathematical meanings. Although it is used to compare distributions, it comes from the field of information theory, where it measures how much "information" is lost when coding a source using a different distribution other than the real one. In information theory, it can be also defined as the difference between entropies - the joint entropy of $Q$ and $P$ and the entropy of $P$.

So to discuss KL divergence, we need to understand the meaning of entropy. The entropy is the measure of "information" in a source, and generally describes how "surprised" you will be with the outcome of the random variable. For instance, if you have a uniform distribution, you will always be "surprised" because there is a wide range of variables it can take. It has high entropy. However, if the RV is a coin with $p=0.9$, then you will probably not be surprised, because it will succeed 90% of the times, so it has low entropy.

Entropy is defined as $H(X)=-\sum_x P(x)\log P(x)=E[-\log P(X)]$, which is the expectation of $I(X)$, the information of a source. Why the log? one reason is the logarithm property of $\log(xy)=\log(x)+\log(y)$, meaning the information of a source composed of independent sources ($p(x)=p_1(x)p_2(x)$) will have the sum of their information. This can only happen by using a logarithm.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you so much for your clear explanation. It helps a lot! $\endgroup$
    – Ralph B.
    Commented Oct 13, 2014 at 13:47
2
$\begingroup$

In short, because Shannon entropy uses logarithm, see:

KL-divergence is typically defined as cross entropy minus entropy.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.