3
$\begingroup$

I'm learning about forward stepwise and there are some things which are not so clear:

  1. If I have $p$ predictors, is it true that forward stepwise does $p$ iterations?

  2. If I add the predictors in each iteration is it true that I eventually converge to the OLS solution? because then I have $p$ predictors added.

  3. Does forward stepwise drop predictors? e.g. If I choose $x_1$ as the best predictor. In the next iteration I compare $x_1$ with all the others, $x_2,x_3,...x_{p-1}$ and I choose the one with the smallest RSS, say $x_2$. Then I compare $x_1,x_2$ based on some criteria, and if that criteria is not fulfilled I drop $x_1$?

What are some of the drawbacks of using forward stepwise?

Improve this question
$\endgroup$
4
  • 6
    $\begingroup$ There are many, many drawbacks to using forward, backwards, all-possible, and AIC-based stepwise regression. Avoid it. Much written on this site. Stepwise can sometimes be OK if using penalization (shrinkage) but even then is problematic. The chance that the data can tell you which variables are "the variables" is zero. $\endgroup$
    – Frank Harrell
    Commented Jun 9, 2018 at 18:44
  • $\begingroup$ @FrankHarrell I'm not using it, I have just to explain to my professor how it works and what are the drawbacks. I'm using lasso. $\endgroup$
    – Ville
    Commented Jun 9, 2018 at 18:47
  • 4
    $\begingroup$ Lasso has low probability of finding the right variables. But better than ordinary stepwise. Problems with ordinary stepwise summarized here: stata.com/support/faqs/statistics/stepwise-regression-problems $\endgroup$
    – Frank Harrell
    Commented Jun 9, 2018 at 18:49
  • $\begingroup$ So what's the alternative? $\endgroup$
    – Mox
    Commented Apr 6, 2020 at 21:19

1 Answer 1

Reset to default
4
$\begingroup$

+1 to @FrankHarrell's comment. That said, there is no "canonical" stepwise algorithm. Implementations may differ.

  1. Usually, the algorithm stops when no more predictors are significant, which may well happen after fewer than $p$ iterations. Alternatively, it may consider entire groups of predictors and add more than one predictor in each step.

  2. See 1.

  3. Some stepwise implementations may add or remove predictors, but that is not forward stepwise any more. (Nor backward stepwise, which starts from the full model and deletes predictors, without adding any back in.)

The drawbacks of (any kind of) stepwise variable selection have been discussed many times here. Best to search within the tag, or look through the top-voted threads.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.