# What is the relationship, if any, between Stein's Paradox and linear restrictions in regressions?

Suppose we have

$$y = b_1x_1 + b_2x_2 + b_3x_3 + e$$

as our regression model.

Setting a linear restriction, say $b_1 + b_2 + b_3 = 0$, allow us to rewrite the model as,

$$y = (b_1)(x_1 - x_3) + (b_2)(x_2 - x_3) + e$$

Given that such restrictions often improve out of sample performance of estimates (even when the restrictions are wrong), I was wondering if it may be connected to Stein's Paradox? Intuitively, it seems to me that $b_3$ is essentially 'shrunk' to zero with the restriction, and that somehow improves the estimates.

I was wondering if anyone could give a more theoretically robust explanation? Or if I am wrong here, point out where I am wrong.

• It will be nice, if somebody can relate it to Stein's Paradox even in the absence of a restriction but when 3 or more coefficients to be estimated. – Cagdas Ozgenc Mar 2 '15 at 13:21
• Restrictions such as $b_1 + b_2 + b_3 = 0$ often improve out of sample performance? Is that really true? I never encountered such a regularization strategy. Instead, what is very often used is a penalty term such as e.g. $\lambda\sum b_i^2$ that needs to be minimized. This particular term leads to "ridge regression", which does indeed shrink all $b_i$ to zero. The relation of ridge regression to Stein's paradox was extensively discussed in this thread. – amoeba Mar 2 '15 at 13:40
• @amoeba Here's a recent paper showing how restrictions helped improved out of sample predictions for the stock market. There are many other such instances in econometric literature. Thanks for the link. I have looked at that before posting this. But I thought it was a bit different. I mean intuitively they seem to be about the same thing, but I am wondering if there is a more solid proof or demonstration. – shellsnail Mar 2 '15 at 13:41
• Thanks for the link, but I don't have time to read this paper now, and I anyway know nothing about stock markets. However, I remain quite skeptical about the word "often". Classical texts in machine learning, such as e.g. The Elements of Statistical Learning by Hastie et al. don't mention such a regularization approach, if I remember correctly. Moreover, it does not even fit a standard scheme with introducing a penalty term into the cost function: a linear constraint is not a "soft" penalty term, but a hard constraint... It looks weird to me. – amoeba Mar 2 '15 at 13:50
• Also, why do you think it will "shrink" $b_3$ to zero? If $b_3 \approx 0$, then this constraint implies $b_1 \approx -b_2$. Why would this be true? – amoeba Mar 2 '15 at 13:52