46
$\begingroup$

Stein's Example shows that the maximum likelihood estimate of $n$ normally distributed variables with means $\mu_1,\ldots,\mu_n$ and variances $1$ is inadmissible (under a square loss function) iff $n\ge 3$. For a neat proof, see the first chapter of Large-Scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction by Bradley Effron.

This was highly surprising to me at first, but there is some intuition behind why one might expect the standard estimate to be inadmissible (most notably, if $x \sim \mathcal N(\mu,1)$, then $\mathbb{E}\|x\|^2\approx \|\mu\|^2+n$, as outlined in Stein's original paper, linked to below).

My question is rather: What property of $n$-dimensional space (for $n\ge 3$) does $\mathbb{R}^2$ lack which facilitates Stein's example? Possible answers could be about the curvature of the $n$-sphere, or something completely different.

In other words, why is the MLE admissible in $\mathbb{R}^2$?


Edit 1: In response to @mpiktas concern about 1.31 following from 1.30:

$$E_\mu\left(\|z-\hat{\mu}\|^2\right)=E_\mu\left(S\left(\frac{N-2}{S}\right)^2\right)=E_\mu\left(\frac{(N-2)^2}{S}\right).$$

$$\hat{\mu_i} = \left(1-\frac{N-2}{S}\right)z_i$$ so $$E_\mu\left(\frac{\partial\hat{\mu_i}}{\partial z_i} \right)=E_\mu\left( 1-\frac{N-2}{S}+2\frac{z_i^2}{S^2}\right).$$ Therefore we have:

$$2\sum_{i=1}^N E_\mu\left(\frac{\partial\hat{\mu_i}}{\partial z_i} \right)=2N-2E_\mu\left(\frac{N(N-2)}{S}\right)+4E_\mu\left(\frac{(N-2)}{S}\right)\\=2N-E_\mu\frac{2(N-2)^2}{S}.$$

Edit 2: In this paper, Stein proves that the MLE is admissible for $N=2$.

$\endgroup$
  • 4
    $\begingroup$ @mpiktas It isn't as inapplicable as it looks. The situation is similar to an ANOVA after we apply a sufficiency reduction. This hints that the usual ANOVA estimates of the group means are inadmissible provided we are trying to estimate the means of more than 3 groups (which turns out the be true). I would recommend looking at proofs that the MLE is admissible for $N = 1, 2$ and seeing where they fail when trying to extend to $N = 3$ rather than just looking at proofs that Stein's estimator does what it claims to do, which is easy once you actually have the estimator in mind. $\endgroup$ – guy Jul 26 '11 at 15:29
  • 2
    $\begingroup$ ...and know to use Stein's Lemma. I guess it's actually a little less straight forward than I though 6 minutes ago. $\endgroup$ – guy Jul 26 '11 at 15:35
  • 2
    $\begingroup$ I agree. Do you have any good references for that (aside from the original paper). I found Stein's original paper overly computational and was hoping that someone would have developed a different method in the last fifty years. $\endgroup$ – Har Jul 26 '11 at 15:45
  • 2
    $\begingroup$ The proof that I was taught was that of Brown and Hwang from 1983 which uses a method suggested by Blyth from the early 1950's I believe. It is pretty general (more general than Stein's result in that it works for the exponential family) and, I believe, quite different from Stein. But it isn't trivial. $\endgroup$ – guy Jul 26 '11 at 17:57
  • 2
    $\begingroup$ @Har great question! (+1) $\endgroup$ – suncoolsu Jul 28 '11 at 18:46
43
$\begingroup$

The dichotomy between the cases $d < 3$ and $d \geq 3$ for the admissibility of the MLE of the mean of a $d$-dimensional multivariate normal random variable is certainly shocking.

There is another very famous example in probability and statistics in which there is a dichotomy between the $d < 3$ and $d \geq 3$ cases. This is the recurrence of a simple random walk on the lattice $\mathbb{Z}^d$. That is, the $d$-dimensional simple random walk is recurrent in 1 or 2 dimensions, but is transient in $d \geq 3$ dimensions. The continuous-time analogue (in the form of Brownian motion) also holds.

It turns out that the two are closely related.

Larry Brown proved that the two questions are essentially equivalent. That is, the best invariant estimator $\hat{\mu} \equiv \hat{\mu}(X) = X$ of a $d$-dimensional multivariate normal mean vector is admissible if and only if the $d$-dimensional Brownian motion is recurrent.

In fact, his results go much further. For any sensible (i.e., generalized Bayes) estimator $\tilde{\mu} \equiv \tilde{\mu}(X)$ with bounded (generalized) $L_2$ risk, there is an explicit(!) corresponding $d$-dimensional diffusion such that the estimator $\tilde{\mu}$ is admissible if and only if its corresponding diffusion is recurrent.

The local mean of this diffusion is essentially the discrepancy between the two estimators, i.e., $\tilde{\mu} - \hat{\mu}$ and the covariance of the diffusion is $2 I$. From this, it is easy to see that for the case of the MLE $\tilde{\mu} = \hat{\mu} = X$, we recover (rescaled) Brownian motion.

So, in some sense, we can view the question of admissibility through the lens of stochastic processes and use well-studied properties of diffusions to arrive at the desired conclusions.

References

  1. L. Brown (1971). Admissible estimators, recurrent diffusions, and insoluble boundary value problems. Ann. Math. Stat., vol. 42, no. 3, pp. 855–903.
  2. R. N. Bhattacharya (1978). Criteria for recurrence and existence of invariant measures for multidimensional diffusions. Ann. Prob., vol. 6, no. 4, 541–553.
$\endgroup$
  • 2
    $\begingroup$ Actually, something like this is what I hoped to. A connection to another field of mathematics (be it differential geometry or stochastic processes) which shows that the admissibility for $n=2$ was not just a fluke. Great answer! $\endgroup$ – Har Jul 30 '11 at 15:34
  • $\begingroup$ Inspired by your answer, I provided some details and also add a geometric explanation in response to this problem on MO: mathoverflow.net/questions/93745/… $\endgroup$ – Henry.L Apr 11 '17 at 20:08
21
$\begingroup$

@cardinal gave a great answer (+1), but the whole issue remains mysterious unless one is familiar with the proofs (and I am not). So I think the question remains as to what is an intuitive reason that Stein's paradox does not appear in $\mathbb R$ and $\mathbb R^2$.

I find very helpful a regression perspective offered in Stephen Stigler, 1990, A Galtonian Perspective on Shrinkage Estimators. Consider independent measurements $X_i$, each measuring some underlying (unobserved) $\theta_i$ and sampled from $\mathcal N(\theta_i, 1)$. If we somehow knew the $\theta_i$, we could make a scatter plot of $(X_i, \theta_i)$ pairs:

Stein's paradox: regression perspective

The diagonal line $\theta = X$ corresponds to zero noise and perfect estimation; in reality the noise is non-zero and so the points are displaced from the diagonal line in horizontal direction. Correspondinly, $\theta = X$ can be seen as a regression line of $X$ on $\theta$. We, however, know $X$ and want to estimate $\theta$, so we should rather consider a regression line of $\theta$ on $X$ -- which will have a different slope, biased horizontally, as shown on the figure (dashed line).

Quoting from the Stigler's paper:

This Galtonian perspective on the Stein paradox renders it nearly transparent. The "ordinary" estimators $\hat \theta_i^0 = X_i$ are derived from the theoretical regression line of $X$ on $\theta$. That line would be useful if our goal were to predict $X$ from $\theta$, but our problem is the reverse, namely to predict $\theta$ from $X$ using the sum of squared errors $\sum (\theta_i - \hat \theta_i)^2$ as a criterion. For that criterion, the optimum linear estimators are given by the least squares regression line of $\theta$ on $X$, and the James-Stein and Efron-Morris estimators are themselves estimators of that optimum linear estimator. The "ordinary" estimators are derived from the wrong regression line, the James-Stein and Efron-Morris estimators are derived from approximations to the right regression line.

And now comes the crucial bit (emphasis added):

We can even see why $k\ge 3$ is necessary: if $k=1$ or $2$, the least squares line of $\theta$ on $X$ must pass through the points $(X_i, \theta_i)$, and hence for $k=1$ or $2$, the two regression lines (of $X$ on $\theta$ and of $\theta$ on $X$) must agree at each $X_i$.

I think this makes it very clear what is special about $k=1$ and $k=2$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.