Do stepwise regression techniques increase a model's predictive power?

Question

I understand some of the many problems of stepwise regression. However, as an academic endeavor, assume I want to use stepwise regression for a predictive model, and I want to better understand the impacts it may have on performance.

Given a linear model, for example, does performing stepwise regression on the model tend to increase or decrease predictive power of the model when presented with new data?

Are there any theoretical impacts that stepwise regression will have on predictive ability?

Practical experience would be helpful as well; perhaps situations when stepwise regression enhances prediction, and when it doesn't.

Answer 1

There are a variety of problems with stepwise selection. I discussed stepwise in my answer here: Algorithms for automatic model selection. In that answer, I did not primarily focus on the problems with inference, but on the fact that the coefficients are biased (the athletes trying out are analogous to variables). Because the coefficients are biased away from their true values, the out of sample predictive error should be enlarged, ceteris paribus.

Consider the notion of the bias-variance trade-off. If you think of the accuracy of your model as the variance of the prediction errors (i.e., MSE: $1/n\sum (y_i -\hat y_i)^2$), the expected prediction error is the sum of three different sources of variance:
$$\newcommand{\Var}{{\rm Var}} E\big[(y_i -\hat y_i)^2\big] = \Var(\hat f) + \big[{\rm Bias}(\hat f)\big]^2 + \Var(\varepsilon) $$ These three terms are the variance of your estimate of the function, the square of the bias of the estimate, and the irreducible error in the data generating process, respectively. (The latter exists because the data are not deterministic—you will never get predictions that are closer than that on average.) The former two come from the procedure used to estimate your model. By default we might think OLS is the procedure used to estimate the model, but it is more correct to say that stepwise selection over OLS estimates is the procedure. The idea of the bias-variance trade-off is that whereas an explanatory model rightly emphasizes unbiasedness, a predictive model may benefit from using a biased procedure if the variance is sufficiently reduced (for a fuller explanation, see: What problem do shrinkage methods solve?).

With those ideas in mind, the point of my answer linked at the top is that a great deal of bias is induced. All things being equal, that will make out of sample predictions worse. Unfortunately, stepwise selection does not reduce the variance of the estimate. At best, its variance is the same, but it is quite likely to make the variance much worse as well (for example, @Glen_b reports only 15.5% of the times were the right variables even chosen in a simulation study discussed here: Why are p-values misleading after performing a stepwise selection?).

Answer 2

The exact effects will depend on the model and the "truth" which, of course, we can't know. You can look at the effects of stepwise in any particular case by crossvalidating or use a simple train and test approach.