17
$\begingroup$

I am wondering about the different ways that Bayesian and Frequentist statistic connect with each other.

I recalled that the Maximum Likelihood estimate of a parameter $\theta$ is not necessarily an unbiased estimator of that parameter.

That made me wonder: Is the Mean Posterior estimate of $\theta$ an unbiased estimator?

That is,

Does $\phi(x)=E(\theta\mid x)$, imply $E(\phi(x)\mid\theta)=\theta$?

Note that this is indeed a meaningful question, since $\phi(x)$, while it is a Bayesian estimator, is simply a function from the data to the real line and so can also be seen as a classical frequentist estimator.

If this question cannot be answered in general, please assume the prior is uniform.

If not, is there some other Bayesian estimator (i.e. a function from the posterior to $\mathbb R$) that is always an unbiased estimator in the frequentist sense?

$\endgroup$
4
  • 1
    $\begingroup$ Maybe you meant "posterior mean" rather than "mean posterior". $\endgroup$ Dec 18, 2020 at 1:41
  • $\begingroup$ The posterior mean is biased in favor of values having a higher prior probability. $\endgroup$ Dec 18, 2020 at 1:42
  • $\begingroup$ @MichaelHardy Question: what's the difference between "posterior mean" and "mean posterior"? $\endgroup$
    – flow2k
    May 21, 2023 at 23:11
  • $\begingroup$ @flow2k : There's no such thing as "mean posterior" as far as I know. $\endgroup$ May 23, 2023 at 4:37

1 Answer 1

15
$\begingroup$

This is a meaningful question which answer is well-known: when using a proper prior $\pi$ on $\theta$, the posterior mean $\delta^\pi(x) = \mathbb{E}^\pi[\theta|x]$ cannot be unbiased. As otherwise the integrated Bayes risk would be zero: \begin{align*} r(\pi; \delta^\pi) &= \overbrace{\mathbb{E}^\pi\{\underbrace{\mathbb{E}^X[(\delta^\pi(X)-\theta)^2|\theta]}_{\text{exp. under likelihood}}\}}^{\text{expectation under prior}}\\ &= \mathbb{E}^\pi\{\mathbb{E}^X[\delta^\pi(X)^2+\theta^2-2\delta^\pi(X)\theta|\theta]\}\\ &= \mathbb{E}^\pi\{\mathbb{E}^X[\delta^\pi(X)^2+\theta^2]|\theta\}- \mathbb{E}^\pi\{\theta \mathbb{E}^X[\delta^\pi(X)|\theta]\}-\overbrace{\mathbb{E}^X\{\mathbb{E}^\pi[\theta|X]\delta^\pi(X)\}}^{\text{exp. under marginal}}\\ &= \mathbb{E}^\pi[\theta^2]+\underbrace{\mathbb{E}^X[\delta^\pi(X)^2]}_{\text{exp. under marginal}} -\mathbb{E}^\pi[\theta^2]-\mathbb{E}^X[\delta^\pi(X)^2]\\ & = 0 \end{align*} [Notations: $\mathbb{E}^X$ means that $X$ is the random variable to be integrated in this expectation, either under likelihood (conditional on $\theta$) or marginal (integrating out $\theta$) while $𝔼^π$ considers $θ$ to be the random variable to be integrated. Note that $\mathbb{E}^X[\delta^\pi(X)]$ is an integral wrt to the marginal, while $\mathbb{E}^X[\delta^\pi(X)|\theta]$ is an integral wrt to the sampling distribution.]

The argument does not extend to improper priors like the flat prior (which is not uniform!) since the integrated Bayes risk is infinite. Hence, some generalised Bayes estimators may turn out to be unbiased, as for instance the MLE in the Normal mean problem which is also a Bayes posterior expectation under the flat prior. (But there is no general property of unbiasedness for improper priors!)

A side property of interest is that $\delta^\pi(x) = \mathbb{E}^\pi[\theta|x]$ is sufficient in a Bayesian sense, as $$\mathbb{E}^\pi\{\theta|\mathbb{E}^\pi[\theta|x]\}=\mathbb{E}^\pi[\theta|x]$$Conditioning upon $\mathbb{E}^\pi[\theta|x]$ is the same as conditioning on $x$ for estimating $\theta$.

$\endgroup$
12
  • 1
    $\begingroup$ Thank you. Just a quick question, what exactly does $E^X(\cdot)$ mean? does it mean $E(\cdot|X)$? $\endgroup$
    – user56834
    Nov 12, 2017 at 14:24
  • $\begingroup$ Ok. You say that this doesn't apply to flat priors. (why are flat priors not the same as uniform?). Does that mean that in the case of a flat prior, the result I'm looking for holds? $\endgroup$
    – user56834
    Nov 12, 2017 at 14:34
  • 1
    $\begingroup$ @SheridanGrant: the "trüe mean" of the observable $\bar{x}$, $\mu_0$, is an unknown and hence varies within a range of possible. Unbiasedness addresses a frequentist property that must hold for all values of $\mu_0$, not just one. In that sense it is impossible to centre the prior at $\mu_0$. $\endgroup$
    – Xi'an
    Jun 17, 2020 at 17:10
  • 1
    $\begingroup$ The statement "Cannot be unbiased" is unambiguous, it means that the posterior expectation cannot be an unbiased estimator, therefore that there exist $\theta$'s such that $\mathbb{E}^X_\theta[\mathbb{E}^\pi[\Theta|X]] \neq \theta$. $\endgroup$
    – Xi'an
    May 22, 2023 at 12:00
  • 1
    $\begingroup$ The outer expectation is a frequentist one computed wrt the frequentist distribution of $X$ indexed/parameterised by $\theta$. This is why I used $\Theta$ as the inner variate, to stress the different meanings of $\theta$ as the actual parameter behind the data and as the random variable distributed from the posterior. $\endgroup$
    – Xi'an
    May 31, 2023 at 10:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.