1
$\begingroup$

I'm reading the Chapter 7 of Statistical Rethinking (2nd), where the author delves into information theory and model selection.

I think I've grasped the concept of what would be the KL Divergence, and the need of knowing the target probability distribution for its calculus. The author then goes into how we don't necessarily need to know the target distribution, however, to estimate their relative divergence to the target:

To use $D_{KL}$ to compare models, it seems like we would have to know $p$, the target probability distribution. In all of the examples so far, I’ve just assumed that $p$ is known. But when we want to find a model $q$ that is the best approximation to $p$, the “truth,” there is usually no way to access $p$ directly. We wouldn’t be doing statistical inference, if we already knew $p$.

But there’s an amazing way out of this predicament. It helps that we are only interested in comparing the divergences of different candidates, say $q$ and $r$. In that case, most of $p$ just subtracts out, because there is a $E\hspace{1mm}log(p_i)$ term in the divergence of both $q$ and $r$. This term has no effect on the distance of $q$ and $r$ from one another. So while we don’t know where $p$ is, we can estimate how far apart $q$ and $r$ are, and which is closer to the target. It’s as if we can’t tell how far any particular archer is from hitting the target, but we can tell which archer gets closer and by how much.

Until here I've found it pretty straightforward. But then, when the author proposes that...

All of this also means that all we need to know is a model’s average log-probability: $E\hspace{1mm}log(q_i)$ for $q$ and $E\hspace{1mm}log(r_i)$ for $r$. These expressions look a lot like log-probabilities of outcomes you’ve been using already to simulate implied predictions of a fit model. Indeed, just summing the log-probabilities of each observed case provides an approximation of $E\hspace{1mm}log(q_i)$. We don’t have to know the $p$ inside the expectation.

That's when he lost me. Is it really as simple as comparing the average log probability scores distribution of different models as an estimate of the KL divergence? I have not found this association between these concepts anywhere else that I've looked (not using those words, at least).

I like to define my concepts very clearly, and now I would like to know if I could define the "average log probability score" as "An approximation of the KL Divergence between each candidate model and the true distribution," or if I interpreted something wrong.

$\endgroup$
3
  • 1
    $\begingroup$ Does this answer your question? Understanding KL divergence in chapter 7 of Statistical Rethinking $\endgroup$
    – kmm
    Jun 9, 2023 at 1:02
  • $\begingroup$ Actually no... this question seems to refer to understading KL divergence conceptually as an measure of distance between distributions. I understand that concept. What I want to know is if the average log probability score can be interpreted as an approximation of the KL Divergence. $\endgroup$
    – Idervas
    Jun 9, 2023 at 1:48
  • 1
    $\begingroup$ see stfwn.com/articles/mle-is-min-kld $\endgroup$
    – seanv507
    Jun 9, 2023 at 4:11

1 Answer 1

4
$\begingroup$

The average log probability score is equal to the KL divergence up to a constant factor (an entropy term) assuming the expectation is taken with respect to $p$ in your example.

We can write the KL divergence as

\begin{align} D(p || q) = H(p, q) - H(p) \end{align}

where $H(p, q)$ is the cross entropy between $p$ and $q$ and $H(p)$ is the entropy of $p$. The author is referring to the fact that $H(p)$ is a constant and not needed if we want to compare $q$ and $r$. Assuming a discrete random variable,

\begin{align} H(p, q) &= -\sum_x p(x) log q(x) \\ &=-\mathbb{E}_{p(x)} log q(x) \end{align}

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.