I have heard/read that Bayesian and Maximum Likelihood inference can be justified as asymptotically minmizing the KL divergence between the pdf $p(x)$ actually describing the data and the parameterized (approximating) pdf $f(x|\hat{\theta})$, where $\hat{\theta}$ is the MLE or MAP estimate. In other words, the claim is that generically, \begin{align} \hat{\theta} = \arg\min_{\theta} \{\text{KL}\left(p(x)||f(x|\theta )\right) \}. \end{align} I have no trouble seeing this for some special cases (see for instance this excellent post here), but I would be interested in a general proof if one exists. I would also be happy about counterexamples proving that the claim cannot be generally true.

  • 1
    $\begingroup$ These lecture notes (stats200.stanford.edu/Lecture16.pdf) give a sketch of the proof that the KL divergence is minimized by the MLE in general (even for a misspecified model). $\endgroup$ – aleshing Dec 19 '17 at 16:37
  • $\begingroup$ Thanks for this! The proof that Theorem 16.3 relies upon pre-supposes that there exists a unique $\theta$ minimizing the KL. I would be interested in a more informative requirement for the KL-minimization property to hold (i.e., when can we assume that this unique $\theta$ actually exists?) - do you have any sources for that, too? $\endgroup$ – Jeremias K Dec 19 '17 at 16:43
  • $\begingroup$ I don't have any sources for that, I just found that pdf on google since it's a pretty commonly stated result. It does point to some other references though, so I'd probably check those out, and just google around for "MLE minimizes KL divergence misspecified" or something to that effect. $\endgroup$ – aleshing Dec 20 '17 at 0:11
  • $\begingroup$ Bayesian inference is not about finding an estimator but deriving a full posterior. That this posterior concentrates around the MLE is a side property, the Bernstein-von Mises Theorem. $\endgroup$ – Xi'an Dec 21 '17 at 6:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.