Empirical Bayes Estimation

Question

Suppose $X_i$ conditioned on $\mu$ is iid $N(\mu, \sigma^2)$ and $\mu$ is distributed as $N(\mu_0, \tau^2)$. Is there a way to estimate $\tau^2$?

Answer 1

Using a method of moments approximation and the unbiased sample variance approximation for the population variance:

$\operatorname{Var}[X] = \sigma^2 + \tau^2$

Then just solve for $\tau^2$

For proof:

For any compound distribution:

$\operatorname{Var}(X) = \operatorname{E}_G\bigl[\operatorname{Var}_F(X|\theta)\bigr] + \operatorname{Var}_G\bigl(\operatorname{E}_F[X|\theta]\bigr)$

Therefore

$ \begin{align} \operatorname{Var}(X) &= \operatorname{E}[\sigma^2] + \operatorname{Var}[\mu]\\ &= \sigma^2 + \tau^2 \end{align}$

Answer 2

The main idea in empirical bayes estimation is to use the observed data to estimate the prior distribution of the parameters, and then use this prior distribution to update the posterior distribution of the parameters based on the observed data.

Let $X_i|\mu\sim N(\mu, \sigma^2)$ and $\mu \sim N(\mu_0, \tau^2).$
Lets find $f(X)$ by estimation of the mean and the variance (law of total variance, law of total expectation):
$E(X) = E_\mu[E(X|\mu)]] = E_\mu(\mu)=\mu_0$
$Var(X) = E_\mu[V(X|\mu)]]+V_\mu[E(X|\mu)]] = E_\mu(\sigma^2)+V_\mu(\mu) = \sigma^2+\tau^2$

So we know that $X_i \sim N(\mu_0, \;\sigma^2+\tau^2)$. Now we can use the method of moments (or MLE) for estimation of $\tau ^2$:

$E(X^2) = V(X)+E^2(X) = \sigma^2+\tau^2 +\mu_0^2$
$\frac{1}{n}\sum X_i^2 = \sigma^2+\tau^2+\mu_0^2$

$\rightarrow \widehat{\tau^2}=\frac{1}{n}\sum X_i^2-\sigma^2-\mu_0^2$

If $\sigma ^2$ is unknown, you can use the sample variance.

Note that $\tau^2$ can't be negative so $\widehat{\tau^2}=max(\frac{1}{n}\sum X_i^2-\sigma^2-\mu_0^2,0)$.

Actually, in this case, you will get the same answer if you will solve with the MLE method instead of MME.