I have come across these slides (slide # 16 & #17) in one of the online courses. The instructor was trying to explain how Maximum Posterior Estimate(MAP) is actually the solution $L(\theta) = \mathcal{I}[\theta \ne \theta^{*}]$, where $\theta^{*}$ is the true parameter.
Can someone please explain how this follows?
Edit: Added the slides, in case the link goes broken.
From going through the slides you shared, it seems to me that the idea is to explain how MAP estimation can be used to estimate different properties of the posterior, such as mean, mode, and median. I will attempt to explain this in the context of General Bayesian Estimators as presented in Stephen M. Kay's book, Fundamentals of Statistical Signal Processing.
Let's start by considering three types of Risk (i.e., cost functions) associated with estimating the parameter $\theta$:
- $C(e) = e^2$
- $C(e) = |e|$
- $if -\delta < e < \delta, C(e)=0$; else $C(e)=1$
where, $e = \theta - \hat{\theta}$, in which $\hat{\theta}$ is the estimated value and $\theta$ is the true parameter. In Bayesian estimation, the objective is to minimize the expected risk, that is:
$E[C(e)]= \int_X \int_{\theta} C(e)p(X,\theta)d\theta dX = \int_X \left[\int_\theta C(e)p(\theta|X)d\theta\right] p(X)dX$
since we only care about $\theta$, we will focus on the inner integral $\min_{\theta}\int_\theta C(e)p(\theta|X)d\theta$.
Now, depending on which $C(e)$ we choose, the estimator will give us a different property of the posterior. For example, if we choose the first case, $C(e) = e^2$, the minimizing $\theta$ for $\int_\theta C(e)p(\theta|X)d\theta$, is the mean. Since you're question is with regards to the indicator function $I[\hat{\theta}\ne \theta]$, I'll address the third risk mentioned above (which if you think about it for $\delta\rightarrow 0$ is equivalent to using the indicator).
For Case 3 above:
$\int_\theta C(e)p(\theta|X)d\theta = \int_{-\infty}^{\hat{\theta}-\delta}p(\theta|X)d\theta + \int_{\hat{\theta}+\delta}^{\infty}p(\theta|X)d\theta = 1 - \int_{\hat{\theta}+\delta}^{\hat{\theta}+\delta}p(\theta|X)d\theta$
which for $\delta \rightarrow 0$ is minimized when $\hat{\theta}$ corresponds to the mode of the posterior.
In the specific case the parameter space $\Theta$ is finite or countably infinite$$\Theta=\{\theta_1,\theta_2,\ldots\}$$ the posterior loss associated with the indicator loss is equal to the probability of being wrong $\mathbb{P}(\hat{\theta}\ne\theta|x)$ and it is minimised when the posterior probability of being correct $\mathbb{P}(\hat{\theta}=\theta|x)$ is maximised. This means that $\hat{\theta}$ is the mode of the posterior distribution or the MAP.
However, this association of MAP and $0-1$ loss is a "folk theorem" in that it is incorrect in most settings, i.e., it does not hold for continuous parameter spaces where $\mathbb{P}(\hat{\theta}=\theta|x)=0$ for all $\hat{\theta}$'s and it further conflicts with the results of Druihlet and Marin (BA, 2007), who point out that the MAP ultimately depends on the choice of the dominating measure. (Even though the Lebesgue measure is implicitly chosen as the default.)
For instance, Evans and Jang posted an arXiv paper in 2011 where they discuss the connection between MAP, least relative surprise (or maximum profile likelihood) estimators, and loss functions. The core of the matter is that neither MAP estimators, nor MLEs are really justified by a decision-theoretic approach, at least in a continuous parameter space. And that the dominating measure [arbitrarily] chosen on the parameter space impacts the value of the MAP, as demonstrated by Druihlet and Marin in 2007. They start in the finite case with the loss function $$ \mathrm{L}(\theta,d) = \mathbb{I}\{\Psi(\theta) \ne d) / \pi_\Psi(\Psi(\theta)) $$ where they consider the estimation of the transform Ψ(θ) by d, inversely weighted by the marginal prior on this transform. In the special case of the identity transform, this loss function leads to the MLE as the Bayes estimator. In the general case, the Bayes estimator is the maximum profile likelihood estimator (LRSE). However, this loss function does not generalise to countably infinite (and obviously continuous) parameter spaces and in such settings the authors can only provide LRSEs as limits of Bayes procedures. The loss function adopted in the countable case is for instance $$ \mathrm{L}(\theta,d) = \mathbb{I}\{\Psi(\theta) \ne d\} / \max\{\eta,\pi_\Psi(\Psi(\theta))\} $$ with the bound decreasing to zero. In the continuous case, the indicator does not work any longer, thus the choice made by the authors is to discretise the space Ψ(Θ) by a specific choice of a partition of balls whose diameters λ go to zero. In the spirit of Druihlet and Marin, this choice depends on a metric (and further regularity conditions). Furthermore, the LRSE itself $$ \max_{\psi}\pi_\psi(\psi|x)/\pi_\psi(\theta) $$ does depend on the version chosen for the densities (if not on the dominating measure), unless one imposes everywhere the Bayes equality $$ \pi_{\psi}(\psi|x)/\pi_\psi(\theta)=f(x|\psi)/m(x) $$ everywhere, when $$ f(x|\psi)=\int_{\{\theta;\Psi(\theta)=\psi\}}f(x|\theta)\pi(\theta)\mathrm{d}\theta $$ and $$ m(x)=\int f(x|\theta)\pi(\theta)\mathrm{d}\theta $$ in the spirit of our Savage-Dickey paradox paper.
Robert Bassett and Julio Deride arXived a paper in 2016 discussing the position of MAPs within Bayesian decision theory.
“…we provide a counterexample to the commonly accepted notion of MAP estimators as a limit of Bayes estimators having 0-1 loss.”
The authors mention my book The Bayesian Choice stating this property without further precautions and I completely agree to being careless in this regard! The difficulty stands with the limit of the maximisers being not necessarily the maximiser of the limit. The paper includes an example to this effect, with a prior as above, associated with a sampling distribution that does not depend on the parameter. The sufficient conditions proposed therein are that the posterior density is almost surely proper or quasiconcave.
See also an alternate characterisation of MAP estimators by Burger and Lucka as proper Bayes estimators under another type of loss function, albeit a rather artificial one. The authors of this arXived paper start with a distance based on the prior; called the Bregman distance, which may be the quadratic or the entropy distance depending on the prior. Defining a loss function that is a mix of this Bregman distance and of the quadratic distance $$ ||K(\hat u-u)||^2+2D_\pi(\hat u,u) $$ produces the MAP as the Bayes estimator. One might still wonder about the dominating measure but both the loss function and the resulting estimator are clearly dependent on the choice of the dominating measure… (The loss depends on the prior but this is not a drawback per se.)
I will give the summary of the text mentioned about this problem in Chapter 5, Bayesian Statistics, Machine Learning: A probabilistic perspective - by Murphy.
Let us say we observed some data $X$, and we want to comment about the posterior distribution of parameters $p(\theta|X)$. Now, the point estimate of the mode of this posterior distribution, which is widely known as the MAP, has certain drawbacks.
Unlike the mean or median, this is an 'untypical' point, in the sense that it does not consider all the other points while being estimated. In the case of estimating the mean/median, we take all the other points into consideration.
So, as expected, in highly skewed posterior distributions, the MAP(and, by extension, the MLE) does not truly represent the actually posterior.
So, how do we summarize a posterior using a point estimate such as Mean/Median/Mode?
This is where people use decision theory - essentially a Loss function $L(\theta, \hat{\theta})$ which is the loss one incurs if the truth is $\theta$ and $\hat{\theta}$ is our estimate. We can choose a variety of Loss functions and our objective here is to minimize the expected value of Loss function.
If the Loss function $L(\theta, \hat{\theta})$ is set as $\mathbb{I}(\hat{\theta}\ne\theta|x)$, an Indicator function for the all times when we CAN NOT estimate the truth, then minimizing the expected value of Loss function wrt $\theta$ is equal to Maximizing this function $\mathbb{I}(\hat{\theta}=\theta|x)$ wrt $\theta$. From this, it is intuitive to guess that Posterior mode minimizes the expected value of loss function. The details of this calculation can be seen in the above answer.
