I have heard/read that Bayesian and Maximum Likelihood inference can be justified as asymptotically minmizing the KL divergence between the pdf $p(x)$ actually describing the data and the parameterized (approximating) pdf $f(x|\hat{\theta})$, where $\hat{\theta}$ is the MLE or MAP estimate. In other words, the claim is that generically, \begin{align} \hat{\theta} = \arg\min_{\theta} \{\text{KL}\left(p(x)||f(x|\theta )\right) \}. \end{align} I have no trouble seeing this for some special cases (see for instance this excellent post here), but I would be interested in a general proof if one exists. I would also be happy about counterexamples proving that the claim cannot be generally true.

  • 1
    $\begingroup$ These lecture notes (stats200.stanford.edu/Lecture16.pdf) give a sketch of the proof that the KL divergence is minimized by the MLE in general (even for a misspecified model). $\endgroup$
    – aleshing
    Commented Dec 19, 2017 at 16:37
  • $\begingroup$ Thanks for this! The proof that Theorem 16.3 relies upon pre-supposes that there exists a unique $\theta$ minimizing the KL. I would be interested in a more informative requirement for the KL-minimization property to hold (i.e., when can we assume that this unique $\theta$ actually exists?) - do you have any sources for that, too? $\endgroup$
    – Jeremias K
    Commented Dec 19, 2017 at 16:43
  • $\begingroup$ I don't have any sources for that, I just found that pdf on google since it's a pretty commonly stated result. It does point to some other references though, so I'd probably check those out, and just google around for "MLE minimizes KL divergence misspecified" or something to that effect. $\endgroup$
    – aleshing
    Commented Dec 20, 2017 at 0:11
  • $\begingroup$ Bayesian inference is not about finding an estimator but deriving a full posterior. That this posterior concentrates around the MLE is a side property, the Bernstein-von Mises Theorem. $\endgroup$
    – Xi'an
    Commented Dec 21, 2017 at 6:51
  • $\begingroup$ The above link is broken. I've sketched the proof here: en.wikipedia.org/wiki/… $\endgroup$
    – Tal Galili
    Commented Jan 16, 2020 at 13:45


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.