# Different versions of forward stepwise regression

Forward stepwise regression is a popular method but I found at least three different versions. I was wondering which one is the most popular, and which one is implemented in R.

1. Let $p$ be the total number of covariates. One version states that, given $k$ covariates have been selected, find the one from $p-k$ covariates that are most correlated with the residual. Then include this variable and regress the old residual on this new variable, yielding a new residual. Meanwhile keep all the original $k$ fitted coefficients the same.

2. Another version differs from the first version in that, after the new variable is included, fit the response $\hat{y}$ on the $k+1$ variables. In other words, the $k$ previously included covariates have coefficients updated too.

3. I also encountered a version as follows. For each of the $p-k$ covariates, consider the augmented model formed by adding this covariate to the $k$ original covariates. Then regress the response $y$ on the {\em $k+1$} covariates. Among these $p-k$ least-squares fits, select the covariate which yields the best fit. Note that this is quite computationally intensive.

• From your description, I can't see how #1 and #3 differ. All the p-k IVs are tested one by one and the one which gives the greatest increase in R-sq is selected. #2, I didn't understand what you meant. – ttnphns Feb 17 '17 at 6:24

## 1 Answer

Following up on @ttnphns: it sounds like in #1 & #3, the coefficients on the $k$ original predictors are not updated, correct? This is not a version of stepwise regression I have ever seen. Every time I see stepwise regression, all coefficients are updated.

And then, as @ttnphns notes, there is no difference between choosing the new covariate that is most strongly correlated with the residuals and choosing the one that yields the largest increase in $R^2$.

(You do know that inferential statistics and are invalid after unless you correct in non-trivial ways for the stepwise variable selection, right?)

• Hi Stephen, in #1 the original coefficients are not updated, whereas in #3 they are updated. – John Wong Feb 17 '17 at 15:47