Empirical Bayes: method of moments

The model for the data is $X_{i}$ ~ $Bin(n_{i},\theta_{i})$ (iid, $i=1,...,k$). The prior distribution is $\theta_{i}$ ~ $Beta(\alpha,\beta)$.

How do we choose (and deduct) moment estimators for $\alpha$ and $\beta$ in this case ?

First, note that the combination of a Binomial distribution for $X_i | n_i, \theta_i$ and a Beta distribution for $\theta_i | \alpha, \beta$ leads directly to a Beta-Binomial distribution for $X_i | n_i, \alpha, \beta$.

The first two moments of the Beta-Binomial distribution are:

$$\mathbb{E}X_i = n_i {\alpha \over \alpha+\beta}$$ $$\sigma^2(X_i) = n_i{(\alpha+\beta+n_i)\alpha \beta \over (\alpha+\beta)^2(\alpha+\beta+1)}$$

The variance can be rewritten as follows: $$\sigma^2(X_i) = {n_i(\alpha+\beta)\alpha \beta \over (\alpha+\beta)^2(\alpha+\beta+1)} + {n_i^2 \alpha \beta \over (\alpha+\beta)^2(\alpha+\beta+1)}$$ $$= {n_i\alpha \beta \over (\alpha+\beta)(\alpha+\beta+1)} + {n_i^2 \alpha \beta \over (\alpha+\beta)^2(\alpha+\beta+1)}$$

Let's define $X = \sum_{i=1}^kX_i$ and $n = \sum_{i=1}^k n_i$. Now, since the $X_i$ are independent, we know that the first two moments of the sum of the $X_i$ are just the sum of the first two moments of the individual $X_i$:

$$\mathbb{E}X = n {\alpha \over \alpha+\beta}$$ $$\sigma^2(X) = {n\alpha \beta \over (\alpha+\beta)(\alpha+\beta+1)} + {\alpha \beta \sum n_i^2 \over (\alpha+\beta)^2(\alpha+\beta+1)}$$

Equating sample moments to the two moments above results in one equation that solves for an estimate $\hat{\theta}$ of the ratio $\theta=\alpha / (\alpha+\beta)$ and another, messier, equation that can be partially written in terms of $\hat{\theta}$, $n$, and $\sum n_i^2$.

$$\hat{\theta} = {\sum X_i \over n}$$ $$s^2(X_i) = {n\hat{\theta}\hat{\beta} +\hat{\theta}(1-\hat{\theta})\sum n_i^2 \over \hat{\alpha}+\hat{\beta}+1}$$

Some more algebra, based upon the relationship $\hat{\beta} = (1-\hat{\theta})\hat{\alpha}/\hat{\theta}$ and substitution, gets us to:

$$s^2(X_i) = {n(1-\hat{\theta})\hat{\alpha} +\hat{\theta}(1-\hat{\theta})\sum n_i^2 \over \hat{\alpha}+(1-\hat{\theta})\hat{\alpha}/\hat{\theta}+1}={n(1-\hat{\theta})\hat{\alpha} +\hat{\theta}(1-\hat{\theta})\sum n_i^2 \over \hat{\alpha}/\hat{\theta}+1}$$ $$=\hat{\theta}(1-\hat{\theta}){n\hat{\alpha} +\hat{\theta}\sum n_i^2 \over \hat{\alpha} + \hat{\theta}}$$ $$=n\hat{\theta}(1-\hat{\theta}){\hat{\alpha} +\hat{\theta}\sum n_i^2/n \over \hat{\alpha} + \hat{\theta}}$$

And a few more steps leads to: $${s^2(X_i) \over n\hat{\theta}(1-\hat{\theta})}\hat{\alpha} + \hat{\theta}=\hat{\alpha} +\hat{\theta}\sum n_i^2/n$$

$$\hat{\alpha} = \hat{\theta}\left(\sum n_i^2/n-1\right)\left({n\hat{\theta}(1-\hat{\theta})\over s^2(X_i)-n\hat{\theta}(1-\hat{\theta})}\right)$$

with some possibility I've messed up the algebra, despite a couple of checks.

Once you've gotten this far, finding $\hat{\beta}$ is straightforward. Note that the MOM estimates won't exist if the sample variance is too small relative to the sample mean! This is because there is a limit on how small the population variance can be; consider all the $\theta_i = \theta$, then the variance of the $X_i$ is just $n_i\theta(1-\theta)$, but the sample variance might be smaller by chance, which would contradict the math that shows that the population variance is larger for any values of $\alpha, \beta$.

• jbowman, am I mistaken, or is $s^2$ defined in terms of $\hat\alpha$, and vice versa? – Christoph Hanck Jan 19 '18 at 14:51
• @ChristophHanck - should have clarified; $s^2(X_i)$ is the sample variance. Since $\hat{\theta}$ is just the sample mean, everything on the r.h.s. of the last equation is from the sample. – jbowman Jan 19 '18 at 15:04