# Intuition behind why Stein's paradox only applies in dimensions $\ge 3$

Stein's Example shows that the maximum likelihood estimate of $n$ normally distributed variables with means $\mu_1,\ldots,\mu_n$ and variances $1$ is inadmissible (under a square loss function) iff $n\ge 3$. For a neat proof, see the first chapter of Large-Scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction by Bradley Effron.

This was highly surprising to me at first, but there is some intuition behind why one might expect the standard estimate to be inadmissible (most notably, if $x\in N(\mu,1)$, then $\mathbb{E}||x||^2\approx ||\mu||^2+n$, as outlined in Stein's original paper, linked to below).

My question is rather: What property of $n$-dimensional space (for $n\ge 3$) does $\mathbb{R}^2$ lack which facilitates Stein's example? Possible answers could be about the curvature of the $n$-sphere, or something completely different.

In other words, why is the MLE admissible in $\mathbb{R}^2$?

Edit 1: In response to mpiktas concern about 1.31 following from 1.30:

(1) $E_\mu(||z-\hat{\mu}||^2)=E_\mu(S(\frac{N-2}{S})^2)=E_\mu(\frac{(N-2)^2}{S})$.

$\hat{\mu_i} = (1-\frac{N-2}{S})z_i$ so $E_\mu(\frac{\partial\hat{\mu_i}}{\partial z_i} )=E_\mu( 1-\frac{N-2}{S}+2\frac{z_i^2}{S^2})$. Therefore we have:

(2) Hence $2\sum_{i=1}^N E_\mu(\frac{\partial\hat{\mu_i}}{\partial z_i} )=2N-2E_\mu(\frac{N(N-2)}{S})+4E_\mu(\frac{(N-2)}{S})=2N-E_\mu\frac{2(N-2)^2}{S}$

Edit 2: In this paper, Stein proves that the MLE is admissible for $N=2$.

(I added tags as well as I could, but surely someone could come up with better tags. If you can and have the permissions, please do add them)

-
I am not familiar with this, but I read the pdf, and some things are quite bizarre. First, we observe one $N$-vector and we want to make inference on $N$ parameters, which is a tad optimistic. Second, the MLE estimate is simply the observation itself, no wonder it does not have the required properties. And third, the equation 1.31 does not follow from the 1.30. In fact I get $E_\mu(N-2)^2/S+2E_\mu(N-2)/S-N$, which is not $N-E_\mu((N-2))^2/S$. And fourthly the last term in the last expression is positive for $N=1$ also so the last statement after the equation 1.31 in the book is ... –  mpiktas Jul 26 '11 at 10:48
... not strictly true. Probably I am missing something trivial, but it would be nice if you added some more context into the question, since it seems that it matters a lot. –  mpiktas Jul 26 '11 at 10:50
It is probably the same narrow place of understanding that OLS is just a best linear unbiased estimator (BLUE), and there exist better biased or nonlinear examples like James-Stein estimator or LASSO estimator that minimizes MSE further. –  Dmitrij Celov Jul 26 '11 at 10:52
@mpiktas It isn't as inapplicable as it looks. The situation is similar to an ANOVA after we apply a sufficiency reduction. This hints that the usual ANOVA estimates of the group means are inadmissible provided we are trying to estimate the means of more than 3 groups (which turns out the be true). I would recommend looking at proofs that the MLE is admissible for $N = 1, 2$ and seeing where they fail when trying to extend to $N = 3$ rather than just looking at proofs that Stein's estimator does what it claims to do, which is easy once you actually have the estimator in mind. –  guy Jul 26 '11 at 15:29
@Har great question! (+1) –  suncoolsu Jul 28 '11 at 18:46

The dichotomy between the cases $d < 3$ and $d \geq 3$ for the admissibility of the MLE of the mean of a $d$-dimensional multivariate normal random variable is certainly shocking.

There is another very famous example in probability and statistics in which there is a dichotomy between the $d < 3$ and $d \geq 3$ cases. This is the recurrence of a simple random walk on the lattice $\mathbb{Z}^d$. That is, the $d$-dimensional simple random walk is recurrent in 1 or 2 dimensions, but is transient in $d \geq 3$ dimensions. The continuous-time analogue (in the form of Brownian motion) also holds.

It turns out that the two are closely related.

Larry Brown proved that the two questions are essentially equivalent. That is, the best invariant estimator $\hat{\mu} \equiv \hat{\mu}(X) = X$ of a $d$-dimensional multivariate normal mean vector is admissible if and only if the $d$-dimensional Brownian motion is recurrent.

In fact, his results go much further. For any sensible (i.e., generalized Bayes) estimator $\tilde{\mu} \equiv \tilde{\mu}(X)$ with bounded (generalized) $L_2$ risk, there is an explicit(!) corresponding $d$-dimensional diffusion such that the estimator $\tilde{\mu}$ is admissible if and only if its corresponding diffusion is recurrent.

The local mean of this diffusion is essentially the discrepancy between the two estimators, i.e., $\tilde{\mu} - \hat{\mu}$ and the covariance of the diffusion is $2 I$. From this, it is easy to see that for the case of the MLE $\tilde{\mu} = \hat{\mu} = X$, we recover (rescaled) Brownian motion.

So, in some sense, we can view the question of admissibility through the lens of stochastic processes and use well-studied properties of diffusions to arrive at the desired conclusions.

References

1. L. Brown (1971). Admissible estimators, recurrent diffusions, and insoluble boundary value problems. Ann. Math. Stat., vol. 42, no. 3, pp. 855–903.
2. R. N. Bhattacharya (1978). Criteria for recurrence and existence of invariant measures for multidimensional diffusions. Ann. Prob., vol. 6, no. 4, 541–553.
-
Actually, something like this is what I hoped to. A connection to another field of mathematics (be it differential geometry or stochastic processes) which shows that the admissibility for $n=2$ was not just a fluke. Great answer! –  Har Jul 30 '11 at 15:34
+1, great answer! –  mpiktas Jul 30 '11 at 18:40
+1 what a great answer. –  G. Jay Kerns Aug 1 '11 at 21:07