Is there anyone can describe me, why this sentence is false? I couldn't get the point from this answer.
$\begingroup$
$\endgroup$
1
-
$\begingroup$ Please add the self-study tag & read its wiki. $\endgroup$– kjetil b halvorsen ♦Commented Dec 15, 2020 at 18:40
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
6
Your posterior is
$$ p(\theta | \mathcal{D}) = \frac{p(\mathcal{D}|\theta)p(\theta)}{p(\mathcal{D})} $$
where $\mathcal{D}$ is your set of data points, $p(\mathcal{D}|\theta)$ is the likelihood and $p(\theta)$ is the prior for the parameter(s) $\theta$. Imagine a degenerate prior, which assigns probability $1$ to a specific values $\tilde{\theta}$ of the parameters. This prior is always equal to $0$, except at $\theta = \tilde{\theta}$ (in other words, it is a Dirac located at $\tilde{\theta}$). Then your posterior is also going to be $0$ for any $\theta$ different from $\tilde{\theta}$, and will be independent of $\mathcal{D}$.
-
$\begingroup$ This exercise sheet uses a specific choice of prior (a Gaussian one). In that case, the MAP indeed converges towards the MLE. But that is not true for all possible priors, as stated in your question. $\endgroup$ Commented Dec 12, 2020 at 20:16
-
$\begingroup$ Exactly. The question is "is it true for ALL priors ?" and the answer is no: sometimes it works (as in your exercise sheet) but sometimes it does not (see my answer). $\endgroup$ Commented Dec 12, 2020 at 20:19
-
$\begingroup$ The Bernstein-von Mises theorem (en.wikipedia.org/wiki/Bernstein%E2%80%93von_Mises_theorem) states (roughly) that, under some conditions, as the number of data points $s$ increases, the MAP converges to the MLE. But this is only true under some conditions (which are detailed in the article), including the smoothness (ie the continuity) of the prior density. A degenerate prior, as illustrated in your first question, does not validate this assumption, and the theorem is thus not applicable. $\endgroup$ Commented Dec 15, 2020 at 19:14
-
$\begingroup$ The relation between the MAP and the MLE is an interesting topic, here are some resources: stats.stackexchange.com/q/331312/271601 stats.stackexchange.com/q/95898/271601 $\endgroup$ Commented Dec 15, 2020 at 19:14
-
$\begingroup$ Finally, you can also see that if your prior is flat (ie non informative), then the MLE and the MAP are equivalent, for any number of data points (check yourself). This is somehow the opposite of the case you mentioned in your first question, in which the prior was a Dirac. $\endgroup$ Commented Dec 15, 2020 at 19:25