Robustness of MAP estimate

Question

In Bayesian inference, we have a dataset $x$ and assumed to come from a known parameterized distribution with unknown parameters $\theta$. We then seek to maximize the posterior $P(\theta|x)$ in order to estimate $\theta$.

Here, $x$ is treated as fixed. How can we study the robustness of the MAP estimator, $\theta^* = \text{argmax} P(\theta|x)$? I.e. how is the distribution/variance of $x$ used to make statements about how $\theta^*$ changes under different samplings of $x$? I know $\text{Var}(P(\theta|x))$ tells us how "unsure" we are of $\theta^*$ for example.

More specifically, the influence function is a measure of robustness of an estimator. Can this be applied to the MAP?

Answer 1

The posterior distribution is using Bayes' rule $$ p(\theta | x ) = \frac{p(x | \theta) p(\theta)}{p(x)} $$ The uncertainty of $\theta$ is given by $p(\theta | x )$, which, loosely speaking tells you the probability of $\theta$ being equal to certain values. From the posterior distribution, the variance can be computed, which is a reduced measure of how wide the distribution is, but cannot replace the distribution itself in quantifying the uncertainty in $\theta$, expect for simple analytical distributions, such as the normal distribution etc. Typically, if $p(x | \theta)$ is widened, that is, increasing variance in $x$, then $p(\theta | x )$ is also widened.

The MAP is $$ \hat{\theta} = \text{argmax}_\theta p(\theta | x ) = \text{argmax}_\theta p(x | \theta) p(\theta) $$ which is an point estimate of $\theta$. The uncertainty of this point estimate is similar to that of the maximum likelihood estimator.

Both these methods assume that the model $p(x|\theta)$ describing the data is "correct". That is, statements made about the estimates should be: "assuming that the data follow our model, $p(\theta|x)$ is our estimate of the uncertainty in $\theta$". This is not the same as stating that the data is truly described by the model. However, if the data follow some other model, the estimated posterior distribution is typically wider than an estimated posterior distribution with data that follows our assumed model better. But then it is a discussion about appropriate models, which is another topic.

Answer 2

We don't have much choice. First our aim is to estimate the parameters $\theta$, and we know the data $X$. If you say that we are sampling $X_s$ that we may try to increase the number of samples.

If samples are fixed, we can do so called online learning, which means:

• we start from the know prior $\theta$
• we calculate the new posterior based on prior and likelihood
• once we have the posterior we do MLE to estimate new $\hat{\theta}$

$$\hat{\theta}=\arg \max _{\theta} \mathbb P(X_s \mid \theta)$$

Condition in here is prior is conjugate prior of the posterior.

Answer 3

The principle behind influence function is to say what if my distribution comes from a distribution that is not really the initial distribution $P$ but from $P_\varepsilon=(1-\varepsilon)P+\varepsilon Q$ i.e. a mixture between the real distribution P and the outlier distribution $Q$.

You are searching for $$\widehat \theta(X_1,\dots,X_n) = \arg\max_{\theta}\mathbb{P}(X_1,\dots,X_n|\theta) $$ where $X$ follows $P_\varepsilon$ instead of $P$.

Be careful that this information is not encoded in the posterior this is not the same thing, the posterior will assess the variability of the estimator not its robustness.

In this example I am not sure how you would define an influence function but you can define its empirical verion called the sensitivity curve. Let $x \in \mathbb{R}^d$ and consider $$SC(x)=n\left(\widehat\theta(X_1,\dots, X_{n-1}, x)-\widehat\theta(X_1,\dots,X_n)\right) $$ for M-estimators this goes to the influence function as $n$ goes to infinity for instance but you could consider this a measure of robustness already.