Is there anyone can describe me, why this sentence is false? I couldn't get the point from this answer.

enter image description here


1 Answer 1


Your posterior is

$$ p(\theta | \mathcal{D}) = \frac{p(\mathcal{D}|\theta)p(\theta)}{p(\mathcal{D})} $$

where $\mathcal{D}$ is your set of data points, $p(\mathcal{D}|\theta)$ is the likelihood and $p(\theta)$ is the prior for the parameter(s) $\theta$. Imagine a degenerate prior, which assigns probability $1$ to a specific values $\tilde{\theta}$ of the parameters. This prior is always equal to $0$, except at $\theta = \tilde{\theta}$ (in other words, it is a Dirac located at $\tilde{\theta}$). Then your posterior is also going to be $0$ for any $\theta$ different from $\tilde{\theta}$, and will be independent of $\mathcal{D}$.

  • $\begingroup$ This exercise sheet uses a specific choice of prior (a Gaussian one). In that case, the MAP indeed converges towards the MLE. But that is not true for all possible priors, as stated in your question. $\endgroup$ Dec 12, 2020 at 20:16
  • $\begingroup$ Exactly. The question is "is it true for ALL priors ?" and the answer is no: sometimes it works (as in your exercise sheet) but sometimes it does not (see my answer). $\endgroup$ Dec 12, 2020 at 20:19
  • $\begingroup$ The Bernstein-von Mises theorem (en.wikipedia.org/wiki/Bernstein%E2%80%93von_Mises_theorem) states (roughly) that, under some conditions, as the number of data points $s$ increases, the MAP converges to the MLE. But this is only true under some conditions (which are detailed in the article), including the smoothness (ie the continuity) of the prior density. A degenerate prior, as illustrated in your first question, does not validate this assumption, and the theorem is thus not applicable. $\endgroup$ Dec 15, 2020 at 19:14
  • $\begingroup$ The relation between the MAP and the MLE is an interesting topic, here are some resources: stats.stackexchange.com/q/331312/271601 stats.stackexchange.com/q/95898/271601 $\endgroup$ Dec 15, 2020 at 19:14
  • $\begingroup$ Finally, you can also see that if your prior is flat (ie non informative), then the MLE and the MAP are equivalent, for any number of data points (check yourself). This is somehow the opposite of the case you mentioned in your first question, in which the prior was a Dirac. $\endgroup$ Dec 15, 2020 at 19:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.