5
$\begingroup$

I am wondering how to get rigorously from Bayes rule to maximum a posteriori estimation (MAP)? I have searched the web, but everything I find is very unclear.

Would someone explain, whether or not this can be done rigorously or alternatively please suggest some good books or papers that address this question?

$\endgroup$
6
  • $\begingroup$ I'm voting to close this question as off-topic because the question just requests references. Could also say it is opinion-based. $\endgroup$ Jan 1, 2018 at 19:00
  • 3
    $\begingroup$ @MichaelChernick reference requests are not off-topic, we even have the references tag for such questions. $\endgroup$
    – Tim
    Jan 1, 2018 at 20:07
  • $\begingroup$ @Tim You can always cite questions that violate current rules. References can be part of the topic of a question without being the full answer. I know that I answered questions about statistical references say 5 years ago but that doesn't mean the rules did not change. I think that if the question is strictly about references it is or should be off topic. $\endgroup$ Jan 1, 2018 at 20:55
  • 3
    $\begingroup$ @Michael The record indicates this community disagrees with you. See stats.meta.stackexchange.com/questions/950 for the original discussion, stats.meta.stackexchange.com/questions/4748 for the most recent, and also refer to related threads at stats.meta.stackexchange.com/questions/2471 and stats.meta.stackexchange.com/questions/2428. $\endgroup$
    – whuber
    Jan 1, 2018 at 23:31
  • 2
    $\begingroup$ @MichaelChernick The question had some problems of syntax, English, punctuation, order of ideas, bad terminology, logic, etc. Perhaps that was what was bothering you, so I fixed it. Having done that I would only remark that this question may have been broached before stats.stackexchange.com/questions/73313/… $\endgroup$
    – Carl
    Jan 2, 2018 at 5:00

2 Answers 2

4
$\begingroup$

[This is an entry I wrote on my blog when the paper by Basset & Deride appeared.]

Robert Bassett and Julio Deride arXived a paper last year discussing the position of MAPs within Bayesian decision theory. A point I have discussed extensively on my blog.

“…we provide a counterexample to the commonly accepted notion of MAP estimators as a limit of Bayes estimators having 0-1 loss.”

The authors mention my book The Bayesian Choice stating this property without further precautions and I completely agree to being careless in this regard! The difficulty stands with the limit of the maximisers being not necessarily the maximiser of the limit. The paper includes an example to this effect, with a prior as above, associated with a sampling distribution that does not depend on the parameter. The sufficient conditions proposed therein are that the posterior density is almost surely proper or quasi-concave.

This is a neat mathematical characterisation that cleans this “folk theorem” about MAP estimators. And for which the authors are to be congratulated! However, I am not very excited by the limiting property, whether it holds or not, as I have difficulties conceiving the use of a sequence of losses in a mildly realistic case. I rather prefer the alternate characterisation of MAP estimators by Burger and Lucka as proper Bayes estimators under another type of loss function, albeit a rather artificial one.

$\endgroup$
0
$\begingroup$

First of all, permit me to elucidate a couple of terms to find the common ground:

  1. $\delta(x)$ - decision rule
  2. $\gamma(\theta, \delta(x))$ - the loss function
  3. $R(\theta, \delta) = \mathbb{E}_{x} [R(\theta, \delta(x))]$ - the risk function
  4. $r(\delta) = \mathbb{E}_{\theta}[R(\theta, \delta(x))]=\mathbb{E}_{\theta}(\mathbb{E}_{x} [R(\theta, \delta(x))])$ - the Bayes risk

Thus, every decision rule is characterised by a single number and Bayesian decision rule is defined as a minimizer of $r(\delta)$.

Keeping things simple, let us consider the following case:

$$ H_0:\ \theta = \theta_0\\ H_1:\ \theta = \theta_1 $$

where $\theta_0\ne\theta_1$. Hence:

$$ \delta(x)\in\{d_0,d_1\} $$

Now we need to define the loss function:

$$ \begin{cases} \gamma(\theta_0, d_0) = 0\\ \gamma(\theta_0, d_1) = 1 \end{cases} \qquad \begin{cases} \gamma(\theta_1, d_0) = 1\\ \gamma(\theta_1, d_1) = 0 \end{cases} $$

Now, we set the things up to derive Maximum a Posteriori. Given a prior probability $\pi(\theta)$, let us evaluate a posterior probability $\pi(\theta|x)$. Directly applying Bayes' theorem:

$$ \pi(\theta|x) = \cfrac{L(x, \theta)\pi(\theta)}{\int L(x, \theta)\pi(\theta)d\theta} $$

The Bayes risk:

$$ \begin{align} r(\delta) &= \mathbb{E}_{\theta, x} \gamma(\theta, \delta(x))\\ &=\mathbb{E}_{\theta} \left[\mathbb{E}_{x}\left[\gamma(\theta, \delta(x))|\theta \right]\right]\\ &=\mathbb{E}_{x} \left[\mathbb{E}_{\theta}\left[\gamma(\theta, \delta(x))|x \right]\right]\\ &=\int f(x) \mathbb{E}_{\theta}\left[\gamma(\theta, \delta(x))|x \right] dx \tag{1} \end{align} $$

In our case:

$$ \mathbb{E}_{\theta}\left[\gamma(\theta, \delta)|x \right] = \sum_{i=0}^{1} \gamma(\theta_i, \delta) \pi(\theta_i|x) \tag{2} $$

If you are given an $x \in \Omega$ value, you need to make a decision in favour of $H_0$ ($\delta(x) = d_0$) or $H_1$ ($\delta(x) = d_1$). Then:

$$ \delta(x) = d_0: \qquad (2)=\mathbb{E}_{\theta}\left[\gamma(\theta, d_0)|x \right]=1\cdot\pi(\theta_1|x) \tag{3} $$ $$ \delta(x) = d_1: \qquad (2)=\mathbb{E}_{\theta}\left[\gamma(\theta, d_1)|x \right]=1\cdot\pi(\theta_0|x) \tag{4} $$

Recall, that the purpose is to minimize the Bayes risk (1), which reduces to minimization of (2) and finally to comparison of (3) and (4). For example, if:

$$ \pi(\theta_1|x) > \pi(\theta_0|x) \Rightarrow \delta(x) = d_1 $$

That is exactly Maximum a Posteriori. Hope this helps.

$\endgroup$
1
  • 1
    $\begingroup$ The reasoning applies in a discrete decision set, but cannot be unambiguously extended to the continuous case. $\endgroup$
    – Xi'an
    Jan 1, 2020 at 11:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.