Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I am wondering how to get rigorously from Bayes rule to maximum a posteriori estimation (MAP)? I have searched the web, but everything I find is very unclear.
Would someone explain, whether or not this can be done rigorously or alternatively please suggest some good books or papers that address this question?
[This is an entry I wrote on my blog when the paper by Basset & Deride appeared.]
Robert Bassett and Julio Deride arXived a paper last year discussing the position of MAPs within Bayesian decision theory. A point I have discussed extensively on my blog.
“…we provide a counterexample to the commonly accepted notion of MAP estimators as a limit of Bayes estimators having 0-1 loss.”
The authors mention my book The Bayesian Choice stating this property without further precautions and I completely agree to being careless in this regard! The difficulty stands with the limit of the maximisers being not necessarily the maximiser of the limit. The paper includes an example to this effect, with a prior as above, associated with a sampling distribution that does not depend on the parameter. The sufficient conditions proposed therein are that the posterior density is almost surely proper or quasi-concave.
This is a neat mathematical characterisation that cleans this “folk theorem” about MAP estimators. And for which the authors are to be congratulated! However, I am not very excited by the limiting property, whether it holds or not, as I have difficulties conceiving the use of a sequence of losses in a mildly realistic case. I rather prefer the alternate characterisation of MAP estimators by Burger and Lucka as proper Bayes estimators under another type of loss function, albeit a rather artificial one.
First of all, permit me to elucidate a couple of terms to find the common ground:
Thus, every decision rule is characterised by a single number and Bayesian decision rule is defined as a minimizer of $r(\delta)$.
Keeping things simple, let us consider the following case:
$$ H_0:\ \theta = \theta_0\\ H_1:\ \theta = \theta_1 $$
where $\theta_0\ne\theta_1$. Hence:
$$ \delta(x)\in\{d_0,d_1\} $$
Now we need to define the loss function:
$$ \begin{cases} \gamma(\theta_0, d_0) = 0\\ \gamma(\theta_0, d_1) = 1 \end{cases} \qquad \begin{cases} \gamma(\theta_1, d_0) = 1\\ \gamma(\theta_1, d_1) = 0 \end{cases} $$
Now, we set the things up to derive Maximum a Posteriori. Given a prior probability $\pi(\theta)$, let us evaluate a posterior probability $\pi(\theta|x)$. Directly applying Bayes' theorem:
$$ \pi(\theta|x) = \cfrac{L(x, \theta)\pi(\theta)}{\int L(x, \theta)\pi(\theta)d\theta} $$
The Bayes risk:
$$ \begin{align} r(\delta) &= \mathbb{E}_{\theta, x} \gamma(\theta, \delta(x))\\ &=\mathbb{E}_{\theta} \left[\mathbb{E}_{x}\left[\gamma(\theta, \delta(x))|\theta \right]\right]\\ &=\mathbb{E}_{x} \left[\mathbb{E}_{\theta}\left[\gamma(\theta, \delta(x))|x \right]\right]\\ &=\int f(x) \mathbb{E}_{\theta}\left[\gamma(\theta, \delta(x))|x \right] dx \tag{1} \end{align} $$
In our case:
$$ \mathbb{E}_{\theta}\left[\gamma(\theta, \delta)|x \right] = \sum_{i=0}^{1} \gamma(\theta_i, \delta) \pi(\theta_i|x) \tag{2} $$
If you are given an $x \in \Omega$ value, you need to make a decision in favour of $H_0$ ($\delta(x) = d_0$) or $H_1$ ($\delta(x) = d_1$). Then:
$$ \delta(x) = d_0: \qquad (2)=\mathbb{E}_{\theta}\left[\gamma(\theta, d_0)|x \right]=1\cdot\pi(\theta_1|x) \tag{3} $$ $$ \delta(x) = d_1: \qquad (2)=\mathbb{E}_{\theta}\left[\gamma(\theta, d_1)|x \right]=1\cdot\pi(\theta_0|x) \tag{4} $$
Recall, that the purpose is to minimize the Bayes risk (1), which reduces to minimization of (2) and finally to comparison of (3) and (4). For example, if:
$$ \pi(\theta_1|x) > \pi(\theta_0|x) \Rightarrow \delta(x) = d_1 $$
That is exactly Maximum a Posteriori. Hope this helps.
