# Understanding error in bayesian inference

Let us say we have:

1. Data $$X$$
2. Parameter that we are trying to estimate is $$\Theta$$

The Bayesian estimation method is to

1. Assume a prior on $$\Theta$$
2. Sample $$x$$ from $$X$$
3. Use Bayes theorem. Compute the posterior $$\Theta\mid(X=x)$$

Now, in case a point estimate is to be reported then one way is to report lowest mean square estimate $$\hat{\Theta} = E[\Theta \mid X]$$. This is supposed to be the parameter that has lowest mean squared error.

The error being defined as $$\bar{\Theta}=\hat{\Theta} - \Theta$$.

Now, two supposedly intuitive properties of $$\bar{\Theta}$$ are

1. $$E[\bar{\Theta}] = 0$$
2. $$E[\bar{\Theta} \mid X] = 0$$

Whenever I think of expected values I think there is an inherent distribution that the expectation is computed using. What is the distribution over which $$E[\bar{\Theta}] = 0$$ is computed? There are two sources of randomness namely $$X$$ and $$\Theta$$.

Moreover, Expectations should be able to be estimated by iterated sampling. That procedure, I think, would like the following in the case of $$E[\bar{\Theta} \mid X]$$

1. Sample $$x$$ from $$X$$.
2. Compute posterior $$\Theta \mid (X = x)$$
3. Compute $$\hat{\theta} = E[\Theta \mid X = x]$$. The expectation of the posterior computed above
4. Sample $$\theta$$ from posterior $$\Theta \mid (X = x)$$
5. Compute $$\bar{\theta}=\hat{\theta} - \theta$$

If you iterate over these 5 steps large number of times then the average of $$\bar{\theta}$$ would converge to 0

What would that procedure look like for $$E[\bar{\Theta}]$$? My attempt:

1. Sample $$x$$ from $$X$$
2. Sample $$\theta$$ from Prior $$\Theta$$
3. Compute posterior $$\Theta \mid (X = x)$$
4. Compute $$\hat{\theta} = E[\Theta \mid X = x]$$. The expectation of the posterior computed above
5. Compute $$\bar{\theta}=\hat{\theta} - \theta$$

If you iterate over these 5 steps large number of times then the average of $$\bar{\theta}$$ would converge to 0

Am I right?

• This confuses me: "Now, in case a point estimate is to be reported then one way is to report lowest mean square estimate $\hat{\Theta} = E[\Theta \mid X]$. This is supposed to be the parameter that has lowest mean squared error." To what degree is that a statement and to what degree a question? You seem to say that you want to report the posterior mean and make it the subject of the analysis thereafter. Or do you rather want a $X$-MSE minimizer? Or a $\Theta$-MSE minimizer? These three are not always the same. Jun 30 at 21:25
• The $X$-MSE minimizer is sometimes the same as the maximum likelihood estimator $\hat{\Theta}_\mathrm{ML}$ and the posterior mode is the maximum a posteriori estimator $\hat{\Theta}_\mathrm{MAP}$. Jun 30 at 21:32
• MAP makes sense for discrete distributions imo. You want to minimize the probability of being wrong. I don't want to maximize the likelihood of data. That is not Bayesian. I am looking for an estimator that minimises the expected error over $\Theta$ Jun 30 at 21:35
• so when writing MSE, you had $(E[(\hat{Θ}−Θ)^\intercal(\hat{Θ}−Θ)|X]$ in mind? The not so nice things about this are: 1. some dimensions in Θ can be more or less meaningless than others, Distance in parameter space is therefore meaningless. 2. your computations so far only work for the posterior mean. I suggest that we forget about all things MSE, and that you indicate this up in your post. Jun 30 at 22:17