# Understanding KL divergence in chapter 7 of Statistical Rethinking

I'm having a hard time understanding McElreath's explanation of how the KL divergence allows us to decide whether one of two models is closer to the 'real' model.

Here is what McElreath writes on p. 209 of the 2nd edition of Statistical Rethinking:

To use Dkl to compare models, it seems like we would have to know p, the target probability distribution. In all of the examples so far, I've just assumed that p is known. But when we want to find a model q that is the best approximation to p, the "truth", there is usually no way to access p directly. We wouldn't be doing statistical inference, if we already knew p.

But there's an amazing way out of this predicament. It helps that we are only interested in comparing the divergences of different candidates, say q and r. In that case, most of p just subtracts out, because there is a Elog(pi) term in the divergence of both q and r. This term has no effect on the distance of q and r from one another. So while we don't know where p is, we can estimate how far apart q and r are, and which is closer to the target. It's as if we can't tell how far any particular archer is from hitting the target, but we can tell which archer gets closer and by how much

I follow this explanation up to the point where he says "and which is closer to the target". If we don't know the target, estimating how far apart q and r tells us nothing about which one of these is closer to the target. For instance, if q is 2 and r is 10, the target can be 0 or 5 or 100 - the difference between q and r gives us no clue about where that target may be and whether q or r is closer to it.

Update 1:

I'm incorporating insights gleaned from the answer by @dipetkov to ask further questions.

First, if I'm understanding correctly, the formula should be:

*Dkl(p,q) - Dkl(p,r) ~ [$$\Sigma$$logp - $$\Sigma$$logq] - [$$\Sigma$$logp - $$\Sigma$$logr] = -$$\Sigma$$logq + $$\Sigma$$logr

This is different from the answer because it ignores the true probabilities ("it's an estimate of Elog(q), just without the final step of dividing by the number of observations").

The question that arises is McElreath goes on to talk about the log probability score for a particular model from hereon, even though the understanding is that the we need to compare the scores from 2 models as per the formula above.

So which one is it? The sum of the log probabilities of a given model, on its own, without comparing to another model, doesn't seem to provide us with much info, yet that's what's defined as deviance on page 210 - the sum of the log-pointwise-predictive-density multiplied by -2.

Update 2:

The accepted answer and further investigation made me understand that deviance is a measure associated with a single model. Then we compare differences in deviances (and perhaps the standard error of the differences in pointwise log-probability between 2 models as McElreath explains on pg. 227) to gauge the relative performance of a model.

## 1 Answer

To determine whether approximation $$q$$ or $$r$$ is closer to the target $$p$$, we subtract their KL divergence from $$p$$:

\begin{aligned} & D_{\operatorname{KL}}(p,q) - D_{\operatorname{KL}}(p,r) \\ &= \Big[\operatorname{E}_p\log(p) -\operatorname{E}_p\log(q) \Big] - \Big[\operatorname{E}_p\log(p) -\operatorname{E}_p\log(r) \Big] \\ & = - \operatorname{E}_p\log(q) + \operatorname{E}_p\log(r) \end{aligned}

Note: I use the p subscript to indicate that the expectations are with respect to p.

If the difference $$D_{\operatorname{KL}}(p,q) - D_{\operatorname{KL}}(p,r)$$ is positive, then $$r$$ is closer to $$p$$ that $$q$$. (Or equivalently, $$q$$ is further from $$p$$ than $$r$$, ie. $$q$$ a worse approximation of the truth.)

Perhaps the phrasing is a bit imprecise. Here "we can estimate how far apart $$q$$ and $$r$$ are" refers to a very specific sense of distance between $$q$$ and $$r$$ in terms of expectations under $$p$$. So the "distance" between $$q$$ and $$r$$ does tells us something about $$p$$. The practical use of this theory is that we can estimate expectations under $$p$$ without knowing $$p$$ exactly, by averaging over a sample of observations drawn from $$p$$.

Understanding KL divergence requires that you know at least a bit of probability theory and the definition of expectations. I recommend Foundations of Probability Theory, from a book in preparation by Michael Betancourt.

Follow-up: You substituted the expectation under the true distribution p with a summation over a sample drawn from the true distribution p (ie. the observed data). It's important to understand the role which p plays as it is present both in the expectation $$\operatorname{E}$$ and in the summation $$\sum$$.

More explicit notation might help. The expectation is with respect to p, so we can write:

$$\operatorname{E}\log(q) = \int p(x) \log q(x) dx$$

And the dataset of size n is drawn from p, so we can write the log-probability score $$S(q)$$ as:

$$\frac{1}{n}S(q) = \frac{1}{n}\sum_{x_i:x_i \sim p} \log q(x_i)$$

Let's illustrate the computation with the example you provide in a comment:

# p, q and r are probability distributions over a binary space X = {1, 2}.
X <- c(1, 2)
# p is the truth and q,r are the approximations.

# Function to compute the expectation of function f under distribution p
ExactExpectation <- function(f) {
sum(p[X] * f[X])
}
# Function to approximate the expectation of function f under distribution p
ObsrvExpectation <- function(f, n = 1e5, seed = 1234) {
set.seed(seed)
xs <- sample(X, n, prob = p, replace = TRUE)
mean(f[xs])
}

# q is "closer" to p
p <- c(0.3, 0.7)
q <- c(0.4, 0.6)
r <- c(0.8, 0.2)

- ExactExpectation(log(q)) + ExactExpectation(log(r))
#> [1] -0.5610844
- ObsrvExpectation(log(q)) + ObsrvExpectation(log(r))
#> [1] -0.5600452

# r is "closer" to p
p <- c(0.79, 0.21)
q <- c(0.4, 0.6)
r <- c(0.8, 0.2)

- ExactExpectation(log(q)) + ExactExpectation(log(r))
#> [1] 0.3168777
- ObsrvExpectation(log(q)) + ObsrvExpectation(log(r))
#> [1] 0.3180782

• Thanks. Please see update to my post as a result of further insights from your answer. Sep 29, 2022 at 16:00
• Your explanation really helped me and I understand most of this. But I'm still getting stuck on the following: Let's say p1=0.3, p2=0.7; q1=0.4, q2=0.6; r1=0.8, r2=0.2. The estimate of divergence as per the formula at the beginning of your answer would be 1.42-1.83 = -0.41. Q is closer to p and so we see a negative result as you explained. However, what's still confusing me is that let's say p1=0.79, p2=0.21. Now r is much closer to p, but our formula doesn't change and we still would deem q as the better model. I understand how the math makes sense, but not from an intuitive view. Sep 30, 2022 at 12:22
• The result does change when you change p. The crux of the matter remains to understand the role that p plays in this. Sep 30, 2022 at 16:17
• You're using the KL divergence in your updated example. But with the deviance formula, we don't know p - so how does the deviance calculation change when p changes? Sep 30, 2022 at 16:20
• Ok, your examples made clear the point that you were making earlier. The key piece I was missing was that the log probability of q and r WILL change if the underlying distribution changes, since it's based on looking at the data. Thanks! Sep 30, 2022 at 21:39