Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

How does the stepwise regression method work for both direction in R with the step() function.

I would think that one variable will be placed into the model and then another that will improve the measuring criteria and the significance of the older variable gets assessed. If the older variable's coefficient is not significant the variable will be removed and a next variable will be placed into the model and so forth.

I am not a 100% sure if this is how the step function with both do it, but can someone please inform me if this is correct, if not how does the both direction criteria implement the stepwise regression in R with step().

share|improve this question
    
The help ?step explains it uses AIC (by default), not significance. –  Glen_b May 11 at 10:52
    
The code can be examined by simply typing step. As it says in the help, it uses add1 and drop1 each time through ... but unlike that mention in the help it actually does them by calling drop1 (the backward part) and then add1 (the forward part). –  Glen_b May 11 at 11:17
4  
A better question might be "Why does R do stepwise regression without penalization?" (as does every other statistical package). As documented at length on this site, stepwise regression is a mindfield of disasters. What motivated you to want to use statistical significance or AIC to choose your model for you? –  Frank Harrell May 11 at 12:36

1 Answer 1

The underlying procedure is beautifully documented in Chambers & Hastie (eds, 1992; Ch. 6) (contrary to what the help page says) on page 237.

stats::step() with the option direction = 'both' works by comparing the AIC improvements from dropping each candidate variable, and adding each candidate variable between the upper and lower bound regressor sets supplied, from the current model, and by dropping or adding the one variable that leads to the best AIC improvement (smallest AIC).

For example, assume that you are fitting a linear regression model with the upper set of variables $\mathcal{U} = \{X_1, X_2, X_3, X_4, X_5, X_6, X_7\}$, and lower set $\mathcal{L} = \{X_1\}$, and the starting object $\mathcal{S}_0 = \{X_1, X_3\}$, then the potential sets of retained regressors might be something like $$ \begin{align} \mathcal{S}_1 &= \{X_1, X_3, X_6\} &\text{ (add $X_6$) }\\ \mathcal{S}_2 &= \{X_1, X_3, X_6, X_4\} &\text{ (add $X_4$) }\\ \mathcal{S}_3 &= \{X_3, X_6, X_4\} &\text{ (drop $X_1$) }\\ \mathcal{S}_4 &= \{X_3, X_6, X_4, X_7\} &\text{ (add $X_7$) }\\ \mathcal{S}_5 &= \{X_3, X_4, X_7\} &\text{ (drop $X_6$) } \end{align} $$ and so on, till no AIC improvements can be made.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.