Stepwise regression $R^2$: Is it biased or inconsistent?

Question

In his skewering of stepwise regression, Frank Harrell mentions that the usual $R^2$ is biased high (I assume when it is calculated as the usual adjusted $R^2$ with the number of parameters set to be the number of parameters that survived the stepwise selection).

That sounds pretty bad, but we use biased estimators all the time. Ridge regression, for instance, leads to biased parameter estimates, with the hope being that the reduction in variance is so much that the mean squared error is lower, despite the bias.

Thus, a biased estimator is not a dealbreaker.

If the stepwise regression $R^2$ estimate converges toward a high-biased value, however, then that inconsistency seems like a dealbreaker.

So is stepwise regression $R^2$ estimation just biased or also inconsistent?

Answer 1

It's certainly biased, but, unlike some other methods, it's biased in ways that are impossible to estimate. The degree of bias would certainly depend on how many steps were taken in the stepwise procedure, and probably also on the "in" and "out" values, the number of IVs, the sample size, and other things. I don't know of a comprehensive article on this, but it's been a while since I looked.

In say, ridge regression, we can control the degree of bias by adjusting parameters.