Suppose we have

$$y = b_1x_1 + b_2x_2 + b_3x_3 + e$$

as our regression model.

Setting a linear restriction, say $b_1 + b_2 + b_3 = 0$, allow us to rewrite the model as,

$$y = (b_1)(x_1 - x_3) + (b_2)(x_2 - x_3) + e$$

Given that such restrictions often improve out of sample performance of estimates (even when the restrictions are wrong), I was wondering if it may be connected to Stein's Paradox? Intuitively, it seems to me that $b_3$ is essentially 'shrunk' to zero with the restriction, and that somehow improves the estimates.

I was wondering if anyone could give a more theoretically robust explanation? Or if I am wrong here, point out where I am wrong.

  • $\begingroup$ It will be nice, if somebody can relate it to Stein's Paradox even in the absence of a restriction but when 3 or more coefficients to be estimated. $\endgroup$ – Cagdas Ozgenc Mar 2 '15 at 13:21
  • $\begingroup$ Restrictions such as $b_1 + b_2 + b_3 = 0$ often improve out of sample performance? Is that really true? I never encountered such a regularization strategy. Instead, what is very often used is a penalty term such as e.g. $\lambda\sum b_i^2$ that needs to be minimized. This particular term leads to "ridge regression", which does indeed shrink all $b_i$ to zero. The relation of ridge regression to Stein's paradox was extensively discussed in this thread. $\endgroup$ – amoeba Mar 2 '15 at 13:40
  • $\begingroup$ @amoeba Here's a recent paper showing how restrictions helped improved out of sample predictions for the stock market. There are many other such instances in econometric literature. Thanks for the link. I have looked at that before posting this. But I thought it was a bit different. I mean intuitively they seem to be about the same thing, but I am wondering if there is a more solid proof or demonstration. $\endgroup$ – shellsnail Mar 2 '15 at 13:41
  • $\begingroup$ Thanks for the link, but I don't have time to read this paper now, and I anyway know nothing about stock markets. However, I remain quite skeptical about the word "often". Classical texts in machine learning, such as e.g. The Elements of Statistical Learning by Hastie et al. don't mention such a regularization approach, if I remember correctly. Moreover, it does not even fit a standard scheme with introducing a penalty term into the cost function: a linear constraint is not a "soft" penalty term, but a hard constraint... It looks weird to me. $\endgroup$ – amoeba Mar 2 '15 at 13:50
  • $\begingroup$ Also, why do you think it will "shrink" $b_3$ to zero? If $b_3 \approx 0$, then this constraint implies $b_1 \approx -b_2$. Why would this be true? $\endgroup$ – amoeba Mar 2 '15 at 13:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.