# Why does the Bayesian posterior concentrate around the minimiser of KL divergence?

Consider the Bayesian posterior $$\theta\mid X$$. Asymptotically, its maximum occurs at the MLE estimate $$\hat \theta$$, which just maximizes the likelihood $$\operatorname{argmin}_\theta\, f_\theta(X)$$.

All these concepts—Bayesian priors, maximizing the likelihood—sound super principled and not at all arbitrary. There’s not a log in sight.

Yet MLE minimizes the KL divergence between the real distribution $$\tilde f$$ and $$f_\theta(x)$$, i.e., it minimizes

$$KL(\tilde f \parallel f_\theta) = \int_{-\infty}^{+\infty} \tilde f(x) \left[ \log \tilde f(x) - \log f_\theta(x) \right] \, dx$$

Woah—where did these logs come from? Why KL divergence in particular?

Why, for example, does minimizing a different divergence not correspond to the super principled and motivated concepts of Bayesian posteriors and maximizing likelihood above?

There seems to be something special about KL divergence and/or logs in this context. Of course, we can throw our hands in the air and say that’s just how the math is. But I suspect there might be some deeper intuition or connections to uncover.

• You can find some ideas here: stats.stackexchange.com/questions/188903/… – kjetil b halvorsen Feb 13 at 21:49
• @kjetilbhalvorsen The previous title sounded like a duplicate; I apologise. I’ve made an edit, and it should be clear why this question is not a duplicate. – Yatharth Agarwal Feb 15 at 1:48
• The other questions asks, “What is KL divergence, and why isn’t it symmetric?” The answers explain the concept of a divergence, and some info about KL. In contrast, this question asks “Why does the Bayesian posterior concentrate around the minimiser of KL divergence?” Simply explaining how divergences don’t have to be symmetric and explaining KL and stating KL is connected to MLE fails to address the crux of the question here: why among the many possible divergence does KL in particular have a special connection to the Bayesian posterior. Does this make sense? – Yatharth Agarwal Feb 15 at 1:50
• Yes, it makes sense, but there is still a problem. The posterior depends also on the prior, and if that is strong, the posteriorcan have a max away from the mle. But the prior is absent from your question. – kjetil b halvorsen Feb 15 at 8:26
• @kjetilbhalversen I meant asymptotically with more and more IID samples and under the (stringent) conditions under which the prior does not matter asymptotically! – Yatharth Agarwal Feb 15 at 20:01

\begin{equation} \begin{aligned} KL(\tilde{f} \parallel f_\theta) &= \int \limits_{-\infty}^\infty \tilde{f}(x) (\log \tilde{f}(x) - \log f_\theta (x)) \ dx \\[6pt] &= \Bigg( \underbrace{- \int \limits_{-\infty}^\infty \tilde{f}(x) \log f_\theta(x) \ dx}_{H(\tilde{f}, f_\theta)} \Bigg) - \Bigg( \underbrace{- \int \limits_{-\infty}^\infty \tilde{f}(x) \log \tilde{f}(x) \ dx}_{H(\tilde{f})} \Bigg), \\[6pt] \end{aligned} \end{equation}
where $$H(\tilde{f})$$ is the entropy of $$\tilde{f}$$ and $$H(\tilde{f}, f_\theta)$$ is the cross-entropy of the $$\tilde{f}$$ and $$f_\theta$$. Entropy can be regarded as measures of the average rate of produced by a density (thought cross-entropy is a bit more complicated). Minimising the KL divergence for a fixed value $$\tilde{f}$$ (as in the problem you mention) is equivalent to minimising the cross-entropy, and so this optimisation can be given an information-theoretic interpretation.