4
$\begingroup$

I have heard/read that Bayesian and Maximum Likelihood inference can be justified as asymptotically minmizing the KL divergence between the pdf $p(x)$ actually describing the data and the parameterized (approximating) pdf $f(x|\hat{\theta})$, where $\hat{\theta}$ is the MLE or MAP estimate. In other words, the claim is that generically, \begin{align} \hat{\theta} = \arg\min_{\theta} \{\text{KL}\left(p(x)||f(x|\theta )\right) \}. \end{align} I have no trouble seeing this for some special cases (see for instance this excellent post here), but I would be interested in a general proof if one exists. I would also be happy about counterexamples proving that the claim cannot be generally true.

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ These lecture notes (stats200.stanford.edu/Lecture16.pdf) give a sketch of the proof that the KL divergence is minimized by the MLE in general (even for a misspecified model). $\endgroup$
    – aleshing
    Dec 19, 2017 at 16:37
  • $\begingroup$ Thanks for this! The proof that Theorem 16.3 relies upon pre-supposes that there exists a unique $\theta$ minimizing the KL. I would be interested in a more informative requirement for the KL-minimization property to hold (i.e., when can we assume that this unique $\theta$ actually exists?) - do you have any sources for that, too? $\endgroup$
    – Jeremias K
    Dec 19, 2017 at 16:43
  • $\begingroup$ I don't have any sources for that, I just found that pdf on google since it's a pretty commonly stated result. It does point to some other references though, so I'd probably check those out, and just google around for "MLE minimizes KL divergence misspecified" or something to that effect. $\endgroup$
    – aleshing
    Dec 20, 2017 at 0:11
  • $\begingroup$ Bayesian inference is not about finding an estimator but deriving a full posterior. That this posterior concentrates around the MLE is a side property, the Bernstein-von Mises Theorem. $\endgroup$
    – Xi'an
    Dec 21, 2017 at 6:51
  • $\begingroup$ The above link is broken. I've sketched the proof here: en.wikipedia.org/wiki/… $\endgroup$
    – Tal Galili
    Jan 16, 2020 at 13:45

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.