Forward stepwise regression is a popular method but I found at least three different versions. I was wondering which one is the most popular, and which one is implemented in R.

  1. Let $p$ be the total number of covariates. One version states that, given $k$ covariates have been selected, find the one from $p-k$ covariates that are most correlated with the residual. Then include this variable and regress the old residual on this new variable, yielding a new residual. Meanwhile keep all the original $k$ fitted coefficients the same.

  2. Another version differs from the first version in that, after the new variable is included, fit the response $\hat{y}$ on the $k+1$ variables. In other words, the $k$ previously included covariates have coefficients updated too.

  3. I also encountered a version as follows. For each of the $p-k$ covariates, consider the augmented model formed by adding this covariate to the $k$ original covariates. Then regress the response $y$ on the {\em $k+1$} covariates. Among these $p-k$ least-squares fits, select the covariate which yields the best fit. Note that this is quite computationally intensive.

  • 1
    $\begingroup$ From your description, I can't see how #1 and #3 differ. All the p-k IVs are tested one by one and the one which gives the greatest increase in R-sq is selected. #2, I didn't understand what you meant. $\endgroup$ – ttnphns Feb 17 '17 at 6:24

Following up on @ttnphns: it sounds like in #1 & #3, the coefficients on the $k$ original predictors are not updated, correct? This is not a version of stepwise regression I have ever seen. Every time I see stepwise regression, all coefficients are updated.

And then, as @ttnphns notes, there is no difference between choosing the new covariate that is most strongly correlated with the residuals and choosing the one that yields the largest increase in $R^2$.

(You do know that inferential statistics and are invalid after unless you correct in non-trivial ways for the stepwise variable selection, right?)

  • $\begingroup$ Hi Stephen, in #1 the original coefficients are not updated, whereas in #3 they are updated. $\endgroup$ – John Wong Feb 17 '17 at 15:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.