# Do stepwise regression techniques increase a model's predictive power?

I understand some of the many problems of stepwise regression. However, as an academic endeavor, assume I want to use stepwise regression for a predictive model, and I want to better understand the impacts it may have on performance.

Given a linear model, for example, does performing stepwise regression on the model tend to increase or decrease predictive power of the model when presented with new data?

Are there any theoretical impacts that stepwise regression will have on predictive ability?

Practical experience would be helpful as well; perhaps situations when stepwise regression enhances prediction, and when it doesn't.

• I don't get this: you start out by asking us to ignore the problems--which include issues with predictive power--and then you ask whether there are such problems! Why not search our site for the answers? One popular one is at stats.stackexchange.com/questions/20836. – whuber Sep 22 '17 at 18:59
• I want to focus on the issues surrounding predictive power (not p-values, coefficient biases, etc.). Based on your feedback, I will make the phrasing of my question less ambiguous. My search of the site has not yielded answers specific to predictive performance of full models, vs stepwise reduced models. – Underminer Sep 22 '17 at 19:37
• Would "ignore issues of.." include ignoring better alternatives, even within the focus on predictive power? – Matthew Drury Sep 22 '17 at 20:36
• @MatthewDrury I am primarily interested in the effects of stepwise regression. That being said, I'd certainly be interested in results from similar automated methods. – Underminer Sep 22 '17 at 20:39
• In the last three paragraphs, there are three different things ? What is exact problem or goal you want to solve ? – Subhash C. Davar Sep 23 '17 at 6:08

Consider the notion of the bias-variance trade-off. If you think of the accuracy of your model as the variance of the prediction errors (i.e., MSE: $1/n\sum (y_i -\hat y_i)^2$), the expected prediction error is the sum of three different sources of variance:
$$\newcommand{\Var}{{\rm Var}} E\big[(y_i -\hat y_i)^2\big] = \Var(\hat f) + \big[{\rm Bias}(\hat f)\big]^2 + \Var(\varepsilon)$$ These three terms are the variance of your estimate of the function, the square of the bias of the estimate, and the irreducible error in the data generating process, respectively. (The latter exists because the data are not deterministic—you will never get predictions that are closer than that on average.) The former two come from the procedure used to estimate your model. By default we might think OLS is the procedure used to estimate the model, but it is more correct to say that stepwise selection over OLS estimates is the procedure. The idea of the bias-variance trade-off is that whereas an explanatory model rightly emphasizes unbiasedness, a predictive model may benefit from using a biased procedure if the variance is sufficiently reduced (for a fuller explanation, see: What problem do shrinkage methods solve?).
• I hate to be the guy to defend stepwise regression...but I don't think it's really universally the case that stepwise AIC will lead to worse predictions than plugging in all covariates without penalties, especially if $n \approx p$. See here for a simulation in which stepAIC does much, much better than plugging in all covariates. – Cliff AB Sep 25 '17 at 21:43