2
$\begingroup$

In Bayesian inference, we have a dataset $x$ and assumed to come from a known parameterized distribution with unknown parameters $\theta$. We then seek to maximize the posterior $P(\theta|x)$ in order to estimate $\theta$.

Here, $x$ is treated as fixed. How can we study the robustness of the MAP estimator, $\theta^* = \text{argmax} P(\theta|x)$? I.e. how is the distribution/variance of $x$ used to make statements about how $\theta^*$ changes under different samplings of $x$? I know $\text{Var}(P(\theta|x))$ tells us how "unsure" we are of $\theta^*$ for example.

More specifically, the influence function is a measure of robustness of an estimator. Can this be applied to the MAP?

$\endgroup$
5
$\begingroup$

The posterior distribution is using Bayes' rule $$ p(\theta | x ) = \frac{p(x | \theta) p(\theta)}{p(x)} $$ The uncertainty of $\theta$ is given by $p(\theta | x )$, which, loosely speaking tells you the probability of $\theta$ being equal to certain values. From the posterior distribution, the variance can be computed, which is a reduced measure of how wide the distribution is, but cannot replace the distribution itself in quantifying the uncertainty in $\theta$, expect for simple analytical distributions, such as the normal distribution etc. Typically, if $p(x | \theta)$ is widened, that is, increasing variance in $x$, then $p(\theta | x )$ is also widened.

The MAP is $$ \hat{\theta} = \text{argmax}_\theta p(\theta | x ) = \text{argmax}_\theta p(x | \theta) p(\theta) $$ which is an point estimate of $\theta$. The uncertainty of this point estimate is similar to that of the maximum likelihood estimator.

Both these methods assume that the model $p(x|\theta)$ describing the data is "correct". That is, statements made about the estimates should be: "assuming that the data follow our model, $p(\theta|x)$ is our estimate of the uncertainty in $\theta$". This is not the same as stating that the data is truly described by the model. However, if the data follow some other model, the estimated posterior distribution is typically wider than an estimated posterior distribution with data that follows our assumed model better. But then it is a discussion about appropriate models, which is another topic.

$\endgroup$
5
  • $\begingroup$ Yes! Var$(p(\theta|x))$ seems to be the most straightforward answer. However is this the only answer? I'm wondering whether it can be connected to the common robustness tool, the influence function $\endgroup$ – 900edges Mar 16 at 1:54
  • $\begingroup$ Or is the influence function simply a tool to estimate the variance? $\endgroup$ – 900edges Mar 16 at 2:05
  • 1
    $\begingroup$ The most accurate measure of the uncertainty in $\theta$ is given by the distribution $p(\theta|x)$. Loosely speaking, It tells you the probability of $\theta$ being equal to certain values. The variance is a measure of how wide the distribution is, but cannot replace the distribution itself in quantifying the uncertainty in $\theta$, expect for some "simple" analytical distributions, such as the normal distribution etc. $\endgroup$ – hakanc Mar 16 at 9:07
  • $\begingroup$ Have not really worked with the influence function, but seems like a way of detecting outliers. However, assuming the data have outliers comes down to assume mathematical models, which I updated the answer with. $\endgroup$ – hakanc Mar 16 at 9:30
  • $\begingroup$ Makes sense, so I guess the posterior already has robustness encoded into it! Thanks :) $\endgroup$ – 900edges Mar 16 at 13:30
1
$\begingroup$

We don't have much choice. First our aim is to estimate the parameters $\theta$, and we know the data $X$. If you say that we are sampling $X_s$ that we may try to increase the number of samples.

If samples are fixed, we can do so called online learning, which means:

  • we start from the know prior $\theta$
  • we calculate the new posterior based on prior and likelihood
  • once we have the posterior we do MLE to estimate new $\hat{\theta}$

$$\hat{\theta}=\arg \max _{\theta} \mathbb P(X_s \mid \theta)$$

Condition in here is prior is conjugate prior of the posterior.

$\endgroup$
0
$\begingroup$

The principle behind influence function is to say what if my distribution comes from a distribution that is not really the initial distribution $P$ but from $P_\varepsilon=(1-\varepsilon)P+\varepsilon Q$ i.e. a mixture between the real distribution P and the outlier distribution $Q$.

You are searching for $$\widehat \theta(X_1,\dots,X_n) = \arg\max_{\theta}\mathbb{P}(X_1,\dots,X_n|\theta) $$ where $X$ follows $P_\varepsilon$ instead of $P$.

Be careful that this information is not encoded in the posterior this is not the same thing, the posterior will assess the variability of the estimator not its robustness.

In this example I am not sure how you would define an influence function but you can define its empirical verion called the sensitivity curve. Let $x \in \mathbb{R}^d$ and consider $$SC(x)=n\left(\widehat\theta(X_1,\dots, X_{n-1}, x)-\widehat\theta(X_1,\dots,X_n)\right) $$ for M-estimators this goes to the influence function as $n$ goes to infinity for instance but you could consider this a measure of robustness already.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.