I understand some of the many problems of stepwise regression. However, as an academic endeavor, assume I want to use stepwise regression for a predictive model, and I want to better understand the impacts it may have on performance.

Given a linear model, for example, does performing stepwise regression on the model tend to increase or decrease predictive power of the model when presented with new data?

Are there any theoretical impacts that stepwise regression will have on predictive ability?

Practical experience would be helpful as well; perhaps situations when stepwise regression enhances prediction, and when it doesn't.

  • 8
    $\begingroup$ I don't get this: you start out by asking us to ignore the problems--which include issues with predictive power--and then you ask whether there are such problems! Why not search our site for the answers? One popular one is at stats.stackexchange.com/questions/20836. $\endgroup$ – whuber Sep 22 '17 at 18:59
  • 3
    $\begingroup$ I want to focus on the issues surrounding predictive power (not p-values, coefficient biases, etc.). Based on your feedback, I will make the phrasing of my question less ambiguous. My search of the site has not yielded answers specific to predictive performance of full models, vs stepwise reduced models. $\endgroup$ – Underminer Sep 22 '17 at 19:37
  • 1
    $\begingroup$ Would "ignore issues of.." include ignoring better alternatives, even within the focus on predictive power? $\endgroup$ – Matthew Drury Sep 22 '17 at 20:36
  • $\begingroup$ @MatthewDrury I am primarily interested in the effects of stepwise regression. That being said, I'd certainly be interested in results from similar automated methods. $\endgroup$ – Underminer Sep 22 '17 at 20:39
  • $\begingroup$ In the last three paragraphs, there are three different things ? What is exact problem or goal you want to solve ? $\endgroup$ – Subhash C. Davar Sep 23 '17 at 6:08

There are a variety of problems with stepwise selection. I discussed stepwise in my answer here: Algorithms for automatic model selection. In that answer, I did not primarily focus on the problems with inference, but on the fact that the coefficients are biased (the athletes trying out are analogous to variables). Because the coefficients are biased away from their true values, the out of sample predictive error should be enlarged, ceteris paribus.

Consider the notion of the bias-variance trade-off. If you think of the accuracy of your model as the variance of the prediction errors (i.e., MSE: $1/n\sum (y_i -\hat y_i)^2$), the expected prediction error is the sum of three different sources of variance:
$$\newcommand{\Var}{{\rm Var}} E\big[(y_i -\hat y_i)^2\big] = \Var(\hat f) + \big[{\rm Bias}(\hat f)\big]^2 + \Var(\varepsilon) $$ These three terms are the variance of your estimate of the function, the square of the bias of the estimate, and the irreducible error in the data generating process, respectively. (The latter exists because the data are not deterministic—you will never get predictions that are closer than that on average.) The former two come from the procedure used to estimate your model. By default we might think OLS is the procedure used to estimate the model, but it is more correct to say that stepwise selection over OLS estimates is the procedure. The idea of the bias-variance trade-off is that whereas an explanatory model rightly emphasizes unbiasedness, a predictive model may benefit from using a biased procedure if the variance is sufficiently reduced (for a fuller explanation, see: What problem do shrinkage methods solve?).

With those ideas in mind, the point of my answer linked at the top is that a great deal of bias is induced. All things being equal, that will make out of sample predictions worse. Unfortunately, stepwise selection does not reduce the variance of the estimate. At best, its variance is the same, but it is quite likely to make the variance much worse as well (for example, @Glen_b reports only 15.5% of the times were the right variables even chosen in a simulation study discussed here: Why are p-values misleading after performing a stepwise selection?).

  • 3
    $\begingroup$ I hate to be the guy to defend stepwise regression...but I don't think it's really universally the case that stepwise AIC will lead to worse predictions than plugging in all covariates without penalties, especially if $n \approx p$. See here for a simulation in which stepAIC does much, much better than plugging in all covariates. $\endgroup$ – Cliff AB Sep 25 '17 at 21:43
  • 2
    $\begingroup$ Thanks, @CliffAB. I upvoted that long ago, but I had forgotten about it. Your substantive answer suggests that the EDA model is worth taking seriously after replication on a fresh sample, & your prediction model is worth taking seriously after validating it against holdout data. I agree w/ both of those. I will acknowledge that stepwise worked better in your simulation, but I'm sure you agree that the situation was narrowly crafted to favor it. $\endgroup$ – gung - Reinstate Monica Sep 26 '17 at 0:42

The exact effects will depend on the model and the "truth" which, of course, we can't know. You can look at the effects of stepwise in any particular case by crossvalidating or use a simple train and test approach.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.