Numeric example of data for special case of stepwise linear regression Stepwise Regression works as follows if I'm correct:


*

*fit the initial model

*add the variable which has its f-stat larger than a in-threshold and repeat step 2. if there are no candidates to enter - go to step 3

*if there is a variable with f-stat less than a out-threshold, remove this variable and go to step 2. Otherwise, terminate.


It's pretty easy to find sample data where some variables would be added to the model and also the data for which algorithm removes one variable after the forward step.
I've never seen and couldn't find any data that would add a variable after step 2 and 3 were completed. Usually, when on step 3 some variable is removed, there are no candidates to extend the model again.
Could someone provide a sample dataset where this situation would occur?
 A: Actually, the steps you listed are incorrect. Stepwise works in the following way:


*

*Add one variable based on some criteria. By default for linear models, this is the F-statistic or the lowest associated p-value.

*Try to remove one variable from the new model. This is based on the same criteria as above, but the cut-off is often different. If the p-value of any variables exceeds a certain limit, delete the variable with the largest p-value.

*If no more variables can be added to the model, return to (1).


The easiest way to see this in action is to create an initial linear model where the first variable is weak. Off the top of my head, I took an example from R.
data(deug) # French exam data
formula = "Algebra ~ Analysis + Proba + Informatic + Economy 
      + Option1 + Option2 + English + Sport"

model_1 = lm(Algebra ~ Sport, data=deug$tab)
step(model_1, scope=formula, data=deug$tab)

Most of the variables are predictive, but you'll see that Sport is removed in the middle of the run. Once it's removed, another variable is found to be predictive, added in, etc.
