# James-Stein Estimator with unequal variances

Every statement I find of the James-Stein estimator assumes that the random variables being estimated have the same (and unit) variance.

But all of these examples also mention that the JS estimator can be used to estimate quantities with nothing to do with each other. The wikipedia example is the speed of light, tea consumption in Taiwan, and hog weight in Montana. But presumably your measurements on these three quantities would have different "true" variances. Does this present a problem?

This ties into a larger conceptual problem that I don't understand, related to this question: James-Stein estimator: How did Efron and Morris calculate $\sigma^2$ in shrinkage factor for their baseball example? We compute the shrinkage factor $c$ as follows:

$$c = 1 - \frac{(k-3) \sigma^2} {\sum (y - \bar{y})^2}$$

Intuitively, I would think that the $\sigma^2$ term is actually $\sigma^2_i$ -- different for each quantity being estimated. But the discussion in that question talks only about using the pooled variance...

I would really appreciate if anyone could clear up this confusion!

• If the variance is $D = \mbox{diag}(\sigma_1^2, \ldots, \sigma_n^2)$ we can just left-multiply by $D^{-1/2}$ to get back to the James-Stein problem. If $D$ is unknown, but each "observation" in the problem is a sample mean calculated on the basis of $m_i$ observations we can estimate $D$ with some $\hat D$ and hope that we also get a James-Stein situation if we pre-multiply by $\hat D^{-1/2}$ instead. – guy Oct 12 '14 at 21:02
• @guy: this is a sensible suggestion (+1), however this will result in the same shrinkage factor for all variables, whereas one would want to shrink variables differently, depending on their variance/uncertainty. See the answer that I have just posted. – amoeba says Reinstate Monica Nov 4 '14 at 15:17
• @amoeba Sure; I wasn't suggesting my estimator was practical, only that it illustrated why people say the things OP mentioned in his/her second paragraph. – guy Nov 4 '14 at 17:54

This question was explicitly answered in the classical series of papers on James-Stein estimator in the Empirical Bayes context written in the 1970s by Efron & Morris. I am mainly referring to:

1. Efron and Morris, 1973, Stein's Estimation Rule and Its Competitors -- An Empirical Bayes Approach

2. Efron and Morris, 1975, Data Analysis with Stein's Estimator and Its Generalizations

3. Efron and Morris, 1977, Stein's Paradox in Statistics

The 1977 paper is a non-technical exposition that is a must read. There they introduce the baseball batting example (that is discussed in the thread you linked to); in this example the observation variances are indeed supposed to be equal for all variables, and the shrinkage factor $$c$$ is constant.

However, they proceed to give another example, which is estimating the rates of toxoplasmosis in a number of cities in El Salvador. In each city different number of people were surveyed, and so individual observations (toxoplasmosis rate in each city) can be thought of having different variances (the lower the number of people surveyed, the higher the variance). The intuition is certainly that data points with low variance (low uncertainty) don't need to be shrunken as strongly as data points with high variance (high uncertainty). The result of their analysis is shown on the following figure, where this can indeed be seen to be happening:

The same data and analysis are presented in the much more technical 1975 paper as well, in a much more elegant figure (unfortunately not showing the individual variances though), see Section 3:

There they present a simplified Empirical Bayes treatment that goes as follows. Let $$X_i|\theta_i \sim \mathcal N(\theta_i, D_i)\\ \theta_i \sim \mathcal N(0, A)$$ where $$A$$ is unknown. In case all $$D_i=1$$ are identical, the standard Empirical Bayes treatment is to estimate $$1/(1+A)$$ as $$(k-2)/\sum X_j ^2$$, and to compute the a posteriori mean of $$\theta_i$$ as $$\hat \theta_i = \left(1-\frac{1}{1+A}\right)X_i = \left(1-\frac{k-2}{\sum X_j^2}\right)X_i,$$ which is nothing else than the James-Stein estimator.

If now $$D_i \ne 1$$, then the Bayes update rule is $$\hat \theta_i = \left(1-\frac{D_i}{D_i+A}\right)X_i$$ and we can use the same Empirical Bayes trick to estimate $$A$$, even though there is no closed formula for $$\hat A$$ in this case (see paper). However, they note that

... this rule does not reduce to Stein's when all $$D_j$$ are equal, and we instead use a minor variant of this estimator derived in [the 1973 paper] which does reduce to Stein's. The variant rule estimates a different value $$\hat A_i$$ for each city. The difference between the rules is minor in this case, but it might be important if $$k$$ were smaller.

The relevant section in the 1973 paper is Section 8, and it is a bit of a tougher read. Interestingly, they have an explicit comment there on the suggestion made by @guy in the comments above:

A very simple way to generalize the James-Stein rule for this situation is to define $$\tilde x_i = D_i^{-1/2} x_i, \tilde \theta_i = D_i^{-1/2} \theta_i$$, so that $$\tilde x_i \sim \mathcal N(\tilde \theta_i, 1)$$, apply [the original James-Stein rule] to the transformed data, and then transform back to the original coordinates. The resulting rule estimates $$\theta_i$$ by $$\hat \theta_i = \left(1-\frac{k-2}{\sum [X_j^2 / D_j]}\right)X_i.$$ This is unappealing since each $$X_i$$ is shrunk toward the origin by the same factor.

Then they go on and describe their preferred procedure for estimating $$\hat A_i$$ which I must confess I have not fully read (it is a bit involved). I suggest you look there if you are interested in the details.