# Empirical Bayes Estimation

Suppose $X_i$ conditioned on $\mu$ is iid $N(\mu, \sigma^2)$ and $\mu$ is distributed as $N(\mu_0, \tau^2)$. Is there a way to estimate $\tau^2$?

Using a method of moments approximation and the unbiased sample variance approximation for the population variance:

$\operatorname{Var}[X] = \sigma^2 + \tau^2$

Then just solve for $\tau^2$

For proof:

For any compound distribution:

$\operatorname{Var}(X) = \operatorname{E}_G\bigl[\operatorname{Var}_F(X|\theta)\bigr] + \operatorname{Var}_G\bigl(\operatorname{E}_F[X|\theta]\bigr)$

Therefore

\begin{align} \operatorname{Var}(X) &= \operatorname{E}[\sigma^2] + \operatorname{Var}[\mu]\\ &= \sigma^2 + \tau^2 \end{align}

• what if you don't know $\sigma^2$? Commented Sep 29, 2017 at 2:05
• @user795305 ask a new question ;) Commented Mar 5, 2023 at 10:36

The main idea in empirical bayes estimation is to use the observed data to estimate the prior distribution of the parameters, and then use this prior distribution to update the posterior distribution of the parameters based on the observed data.

Let $$X_i|\mu\sim N(\mu, \sigma^2)$$ and $$\mu \sim N(\mu_0, \tau^2).$$
Lets find $$f(X)$$ by estimation of the mean and the variance (law of total variance, law of total expectation):
$$E(X) = E_\mu[E(X|\mu)]] = E_\mu(\mu)=\mu_0$$
$$Var(X) = E_\mu[V(X|\mu)]]+V_\mu[E(X|\mu)]] = E_\mu(\sigma^2)+V_\mu(\mu) = \sigma^2+\tau^2$$

So we know that $$X_i \sim N(\mu_0, \;\sigma^2+\tau^2)$$. Now we can use the method of moments (or MLE) for estimation of $$\tau ^2$$:

$$E(X^2) = V(X)+E^2(X) = \sigma^2+\tau^2 +\mu_0^2$$
$$\frac{1}{n}\sum X_i^2 = \sigma^2+\tau^2+\mu_0^2$$

$$\rightarrow \widehat{\tau^2}=\frac{1}{n}\sum X_i^2-\sigma^2-\mu_0^2$$

If $$\sigma ^2$$ is unknown, you can use the sample variance.

Note that $$\tau^2$$ can't be negative so $$\widehat{\tau^2}=max(\frac{1}{n}\sum X_i^2-\sigma^2-\mu_0^2,0)$$.

Actually, in this case, you will get the same answer if you will solve with the MLE method instead of MME.

• Why set $E_\mu(\mu)=0$? Commented Mar 4, 2023 at 11:31
• you are right. fixed. Commented Mar 5, 2023 at 12:52