12
$\begingroup$

OLS estimator solves the following minimization problem:
$$\min ||y-X\beta||^2.$$ By taking the FOC, we obtain $\hat{\beta}$, which minimizes the objective function.

But the James-Stein estimator achieves a lower MSE. I know the proof for James-Stein, but how do I reconcile it with the fact that squared error loss is already minimized in the OLS estimate? What am I missing here?

And if the James-Stein estimator achieves lower MSE, why do we still use OLS? Although James-Stein requires the distribution to be normal, isn't it still useful as approximate normality can be satisfied according to some version of the Central Limit Theorem?

$\endgroup$
3
  • 4
    $\begingroup$ To what MSE does the James-Stein estimator refer? $\endgroup$
    – Dave
    Jan 28 at 23:45
  • $\begingroup$ @user404474 the fact that you underscore is not a news, read here: en.wikipedia.org/wiki/James%E2%80%93Stein_estimator $\endgroup$
    – markowitz
    Jan 29 at 7:31
  • $\begingroup$ minimize the variance of estimator or minimize the MSE is not the same thing if the estimator is biased and James-Stein estimator is. $\endgroup$
    – markowitz
    Jan 29 at 7:32

2 Answers 2

22
$\begingroup$
  1. The MSE (mean squared error) is a feature of the distribution of the estimator of $\beta$. This is different from $\|y-X\hat\beta\|^2$ computed on the data. The "squared error loss" minimised by the OLS estimator is not the same as the MSE. Note that the term "error" is used in two different ways here. Calling $\|y-X\hat\beta\|^2$ "squared error loss" calls an observed deviation in the fit between the observed $y$ and the fit $X\hat\beta$ "error", whereas the MSE calls the difference between the estimated $\hat\beta$ and the (assumed) true value of $\beta$ an "error". This "error" cannot be observed as the true $\beta$ is not known, and the "M" in MSE refers to the expected value (mean) of the squared estimation error. Therefore there is no contradiction here.

  2. "Why do we still use OLS?" Well, I can't tell why you still use the OLS; chances are nobody is stopping you from using James-Stein. An unsettling feature of James-Stein is that shrinking toward any user-specified value of $\beta$ basically has the same properties, but may result in rather different outcomes, i.e., as user you need to decide toward which value to shrink, and this has an impact, even though you may not have relevant information about which value to choose (but of course you can use 0 as a default if just in order to avoid something that feels like a subjective decision; but doing something in order to avoid a subjective decision is a subjective decision itself). In some situations one could probably also argue that bias is more important than variance, particularly when different interests are at stake and one group of people would have an advantage if an estimated coefficient were too large , another one if it were too small. In such a situation one should probably accept the price of a larger MSE to not become vulnerable to the criticism that estimators were biased in order to favour one interest over another. The James-Stein estimator is biased and will therefore be problematic in such a situation.

  3. It is possible to improve further on the James-Stein estimator regarding the MSE (see comment by @Thomas Lumley), so arguably if you don't want OLS because of inferior MSE, you shouldn't use James-Stein either (I won't address here specific potential disadvantages of using an even "better" estimator).

  4. The MSE is based on the squared difference between $\beta$ and $\hat\beta$, and is therefore dominated by large possible discrepancies between them. I suspect (although I haven't proved it mathematically) that by "shrinking" the estimated parameter values James-Stein (JS) improves on OLS regarding very large discrepancies but can be worse regarding small discrepancies (by systematically "shrinking away" from the true value where the data suggest something close to it). Now one could argue that in many situations if $\beta$ is really badly estimated, it doesn't matter so much how bad $\hat\beta$ exactly is. This suggests that in many problems one should be maybe more interested in the loss $E\|\beta-\hat\beta\|$ than in the loss $E\|\beta-\hat\beta\|^2$, and OLS may beat JS regarding the former but not the latter. Of course (a) this depends on how large the discrepancies are actually expected to be and (b) what consequences estimation errors of a certain size have in the real situation. Furthermore following this argument one may wonder why not to optimise $E\|\beta-\hat\beta\|$ in the first place, which may require yet another estimator. Still I suspect that OLS does better regarding the unsquared loss, and that in several situations this is a valid reason to prefer it to JS.

$\endgroup$
3
  • 2
    $\begingroup$ I'd also add that the James-Stein estimator is not itself admissible - it's dominated by the positive-part James-Stein estimator. And that isn't admissible, either. I think an admissible estimator that dominates the mean is now known, though. $\endgroup$ Jan 29 at 6:17
  • $\begingroup$ @ThomasLumley Interesting! Any further details you can give on that? (I am very tempted to ask a fresh question on this, but it would be nice to know what's "out there" to help me write one that's pushing in the right direction!) $\endgroup$
    – Silverfish
    Jan 29 at 19:01
  • 1
    $\begingroup$ Hi @ThomasLumley , I think admissible estimators that dominate the sample mean have been known since 1971, see jstor.org/stable/2958496 . Kubokawa later showed that one of Strawderman's estimators dominates the JS estimator, see sciencedirect.com/science/article/pii/0047259X9190096K $\endgroup$
    – Ben
    Jan 29 at 19:50
10
$\begingroup$

In addition to the good points in Christian Hennig's answer, here is another angle to consider:

The OLS estimator $$\hat\beta^{OLS}=(X'X)^{-1}X'y$$ minimizes the sum of squares $$\hat y'\hat y=[(X(X'X)^{-1}X-I)y]'[(X(X'X)^{-1}X-I)y]$$ in the training sample. The James-Stein estimator does not do that.

However, what we are usually interested in is the population or the data generating process and a test sample taken from that. OLS is not guaranteed to minimize the corresponding sum of squares there (in a particular instance or even in expectation).

$\endgroup$
1
  • $\begingroup$ (+1) I think this clarification should prove helpful to many readers! From experience, a lot of students seem to lose track of what we are minimising at each stage, and I think that's evident in the OP's question. Something I think would make this point even more useful - but sadly don't know the answer to - is to add a brief comment about what's known about the performance of the James-Stein estimator vs OLS not just in the training sample but in the DGP itself. $\endgroup$
    – Silverfish
    Jan 29 at 19:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.