# Understanding intuitive difference between KL divergence and Cross entropy

I know there are related questions already asked, for example this one.

I also know the following:

• KL divergence $$D_{KL}(P\Vert Q)$$ is given as:

\begin{align} D_{KL}(P\Vert Q) & = -\sum_xP(x)\log\left(\frac{Q(x)}{P(x)}\right) \\ & = \sum_xP(x)\log\left(\frac{P(x)}{Q(x)}\right) \\ & = \sum_xP(x)\log(P(x))-\sum_xP(x)\log(Q(x)) \\ & = -\underbrace{\sum_xP(x)\log\left(\frac{1}{P(x)}\right)}_{\text{Entropy } H(x)}\underbrace{-\sum_xP(x)\log(Q(x))}_{\text{Cross entropy } H(P,Q) } \\ & = -H(X)+H(P,Q) \qquad\qquad...\text{equation(1)}\\ \end{align}

• Cross Entropy is given as

$$H(P,Q)=-\sum_xP(x)\log Q(x)$$

(Please correct me if I am incorrect in above equations.)

Knowing all this, I want to build more precise intuition behind the difference.

Wikipedia defines KL divergence as follows:

KL divergence of P from Q is the expected "excess" surprise from using Q as a model when the actual distribution is P

I believe the word "excess" refers to term $$-H(X)$$ in equation (1) and we can drop it (essentially dropping $$-H(X)$$) to get definition for cross entropy:

Cross entropy of P from Q is the expected "excess" surprise from using Q as a model when the actual distribution is P.

Q1. Am I correct with this?

If we consider a target or underlying probability distribution $$P$$ and an approximation of the target distribution $$Q$$, then the cross-entropy of $$Q$$ from $$P$$ is the number of additional bits required to represent an event using $$Q$$ instead of $$P$$.

I believe this is wrong. Above definition should be of KL divergence and the word additional refers to $$-H(x)$$ term in equation (1). If we drop it (and hence $$-H(x)$$), we get the definition for cross entropy:

• Cross entropy of $$Q$$ from $$P$$ is the number of "additional" bits required to represent an event using $$Q$$ instead of $$P$$.
• KL divergence of $$Q$$ from $$P$$ is the number of "additional" bits required to represent an event using $$Q$$ instead of $$P$$.

Q2. Am I correct with these interpretations / definitions (in terms of bits requirements)?

Update

$$D_{KL}(p\Vert q)$$ measures the average number of extra bits per message, whereas $$H(p,q)$$ measures the average number of total bits per message.