# Stepwise regression in R with both direction

How does the stepwise regression method work for both direction in R with the step() function.

I would think that one variable will be placed into the model and then another that will improve the measuring criteria and the significance of the older variable gets assessed. If the older variable's coefficient is not significant the variable will be removed and a next variable will be placed into the model and so forth.

I am not a 100% sure if this is how the step function with both do it, but can someone please inform me if this is correct, if not how does the both direction criteria implement the stepwise regression in R with step().

-
The help ?step explains it uses AIC (by default), not significance. –  Glen_b May 11 '14 at 10:52
The code can be examined by simply typing step. As it says in the help, it uses add1 and drop1 each time through ... but unlike that mention in the help it actually does them by calling drop1 (the backward part) and then add1 (the forward part). –  Glen_b May 11 '14 at 11:17
A better question might be "Why does R do stepwise regression without penalization?" (as does every other statistical package). As documented at length on this site, stepwise regression is a mindfield of disasters. What motivated you to want to use statistical significance or AIC to choose your model for you? –  Frank Harrell May 11 '14 at 12:36

stats::step() with the option direction = 'both' works by comparing the AIC improvements from dropping each candidate variable, and adding each candidate variable between the upper and lower bound regressor sets supplied, from the current model, and by dropping or adding the one variable that leads to the best AIC improvement (smallest AIC).
For example, assume that you are fitting a linear regression model with the upper set of variables $\mathcal{U} = \{X_1, X_2, X_3, X_4, X_5, X_6, X_7\}$, and lower set $\mathcal{L} = \{X_1\}$, and the starting object $\mathcal{S}_0 = \{X_1, X_3\}$, then the potential sets of retained regressors might be something like \begin{align} \mathcal{S}_1 &= \{X_1, X_3, X_6\} &\text{ (add X_6) }\\ \mathcal{S}_2 &= \{X_1, X_3, X_6, X_4\} &\text{ (add X_4) }\\ \mathcal{S}_3 &= \{X_3, X_6, X_4\} &\text{ (drop X_1) }\\ \mathcal{S}_4 &= \{X_3, X_6, X_4, X_7\} &\text{ (add X_7) }\\ \mathcal{S}_5 &= \{X_3, X_4, X_7\} &\text{ (drop X_6) } \end{align} and so on, till no AIC improvements can be made.