2
$\begingroup$

I know there are related questions already asked, for example this one.

I also know the following:

  • KL divergence $D_{KL}(P\Vert Q)$ is given as:

$$\begin{align} D_{KL}(P\Vert Q) & = -\sum_xP(x)\log\left(\frac{Q(x)}{P(x)}\right) \\ & = \sum_xP(x)\log\left(\frac{P(x)}{Q(x)}\right) \\ & = \sum_xP(x)\log(P(x))-\sum_xP(x)\log(Q(x)) \\ & = -\underbrace{\sum_xP(x)\log\left(\frac{1}{P(x)}\right)}_{\text{Entropy } H(x)}\underbrace{-\sum_xP(x)\log(Q(x))}_{\text{Cross entropy } H(P,Q) } \\ & = -H(X)+H(P,Q) \qquad\qquad...\text{equation(1)}\\ \end{align} $$

  • Cross Entropy is given as

$$H(P,Q)=-\sum_xP(x)\log Q(x)$$

(Please correct me if I am incorrect in above equations.)

Knowing all this, I want to build more precise intuition behind the difference.

Wikipedia defines KL divergence as follows:

KL divergence of P from Q is the expected "excess" surprise from using Q as a model when the actual distribution is P

I believe the word "excess" refers to term $-H(X)$ in equation (1) and we can drop it (essentially dropping $-H(X)$) to get definition for cross entropy:

Cross entropy of P from Q is the expected "excess" surprise from using Q as a model when the actual distribution is P.

Q1. Am I correct with this?

Also, this article defines cross entropy as follows:

If we consider a target or underlying probability distribution $P$ and an approximation of the target distribution $Q$, then the cross-entropy of $Q$ from $P$ is the number of additional bits required to represent an event using $Q$ instead of $P$.

I believe this is wrong. Above definition should be of KL divergence and the word additional refers to $-H(x)$ term in equation (1). If we drop it (and hence $-H(x)$), we get the definition for cross entropy:

  • Cross entropy of $Q$ from $P$ is the number of "additional" bits required to represent an event using $Q$ instead of $P$.
  • KL divergence of $Q$ from $P$ is the number of "additional" bits required to represent an event using $Q$ instead of $P$.

Q2. Am I correct with these interpretations / definitions (in terms of bits requirements)?

Update

I came across this awesome answer, and after reading it, I guess I was correct in about interpretations. The answer states:

$D_{KL}(p\Vert q)$ measures the average number of extra bits per message, whereas $H(p,q)$ measures the average number of total bits per message.

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.