This is an extension to this previous question, and is related to exercise 4.7 from Gelman et al.'s BDA3.

When is the Bayesian posterior mean $m(y) \equiv E[\theta \mid y]$ unbiased for $\theta$, considering theta either as a random variable (Bayesian perspective) or fixed (Frequentist perspective)?

| cite | improve this question | | | | |

Take the Bayesian perspective, supposing that $E[m(y)\mid\theta] = \theta$ (posterior mean is unbiased). Then

$$E[\theta m(y)] = E[E[\theta m(y) \mid \theta]] = E[\theta E[m(y)\mid\theta]] = E[\theta^2].$$

But also

$$E[\theta m(y)] = E[E[\theta m(y) \mid y]] = E[m(y) E[\theta\mid y]] = E[m(y)^2].$$


$$E[(m(y)-\theta)^2] = E[m(y)^2] + E[\theta^2] - 2E[\theta m(y)] = 0$$

which implies that $m(y) = \theta$ almost surely. This can occur when, e.g., the prior is a point mass at $\mu_0$.

Take the frequentist perspective, where $\theta = \mu_0$ and unbiasedness means

$$E[m(y)\mid\theta] = E[m(y)\mid\mu_0] = E[m(y)] = \mu_0.$$

Then one can construct a prior that yields an unbiased posterior, e.g. if $y_i \sim_\text{iid} N(\theta, \sigma^2)$ and we suppose $\theta \sim N(\mu_0, \sigma_0^2)$ then the posterior mean is the well-known weighted average

$$m(y) = \frac{\sigma_0^2}{\sigma_0^2 + \sigma^2/n}\bar{y} + \frac{\sigma^2/n}{\sigma_0^2 + \sigma^2/n}\mu_0$$

which has expectation $\mu_0$ when the truth is $\theta = \mu_0$. Here $m(y) \ne \theta = \mu_0$ almost surely.

However, in this setting the model is wrong (specifically, the prior). The true prior would be a point mass at $\mu_0$, giving us $m(y) = \mu_0$, which aligns with the Bayesian perspective. Thus, the Bayesian model posterior mean may be nondegenerate and unbiased when the model is wrong!

| cite | improve this answer | | | | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.