Assume I have $N$ samples $x_1, \cdots, x_N$ from a Gaussian random variable $X\sim N(\mu, \sigma^2)$ where both $\mu$ and $\lambda = 1/\sigma^2$ are unknown.

If I apply MLE, I have $\mu_{MLE} = \frac{1}{N} \sum_i x_i$, and $\lambda_{MLE}^{-1} = \frac{1}{N} \sum_i (x_i - \mu_{MLE})^2$. Note that $\lambda_{MLE}^{-1}$ is biased, and the unbiased estimate will be $\frac{N}{N-1} \lambda_{MLE}^{-1}$.

Now, I have a Gamma distribution $Gamma(a_0, b_0)$ as a prior for $\lambda$.

With $N$ samples, we have that the posterior of $\lambda$ is $Gamma(a_N, b_N)$ where $a_N = a_0 + N/2$ and $b_N = b_0 + N/2 \lambda_{MLE}^{-1}$.

Obviously, if I choose $a_0=0$ and $b_0=0$, then I reach the MLE solution, which is biased.

My question is, is there any prior for $\lambda$ that gives me an unbiased estimate $\frac{N}{N-1} \lambda_{MLE}^{-1}$? It seems that $a_0=-1/2, b_0=0$ does the job, but that does not define a proper Gamma distribution.


This is a general result of (Blackwell and Girshik, 1954): There is no (proper) prior distribution leading to the posterior expectation being an unbiased estimator.

proof (Blackwell and Girshik, 1954): Assume that the posterior expectation (and Bayes estimate under the squared error loss) $$ \delta(x) = \mathbb{E}[\theta|X=x] $$ is unbiased for $\theta$, i.e. satisfies $$ \mathbb{E}_{\theta_0}[\delta(X)]=\theta_0 $$ for every value of $\theta_0$ (where the expectation is under the conditional distribution of $X$ given $\theta=\theta_0$. Then, by Fubini's theorem, using the joint distribution of $(\theta,X)$, $$ \mathbb{E}[\theta \delta(X)] = \mathbb{E}[\mathbb{E}[\theta|X]\delta(X)] = \mathbb{E}[\delta(X)^2] $$ if we first integrate in $\theta$ and $$ \mathbb{E}[\theta \delta(X)] = \mathbb{E}[\theta\mathbb{E}[\delta(X)|\theta]] = \mathbb{E}[\theta^2]. $$ if we first integrate in $X$. Therefore, the Bayes risk is null, $$ \mathbb{E}[\{\theta - \delta(X)\}^2] = 0, $$ which is impossible (except in the uninteresting settings where the true value of the parameter can be known for sure). $\blacksquare$

Naturally, if we remove the condition that the prior is proper, there are many instances of the contrary: it is possible to derive unbiased estimators as posterior expectations with improper priors, as shown in the question for $a_0=-1/2$ and $b_0=0$.

| cite | improve this answer | |
  • 1
    $\begingroup$ In the last comment, do you mean $a_0=-1/2, b_0=0$? Because $a_0=0, b_0=0$ leads to a biased estimate for variance. $\endgroup$ – calbear Oct 22 '12 at 0:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.