How does "stepwise regression" work?

Question

I used the following R code to fit a probit model:

p1 <- glm(natijeh ~ ., family=binomial(probit), data=data1)
stepwise(p1, direction='backward/forward', criterion='BIC')

I want to know what does stepwise and backward/forward do exactly and how select the variables?

Answer 1

Principle of stepwise selection

1. You fit a model with all variables you wish. This is your current best model.
2. You remove one variable (or add one, among variable not used in the current best model), and for each one, you fit the new model, and you compare them with each over and with the original one, according to BIC (or any other criterion, such as AIC). You get another "current best model".

You repeat 2. until there no reduction of BIC. You have only a local minimum of BIC, which means you may not get the best model among all possible choices of subsets of variables. But anyway, there are usually too many of them, so this is a way to optimize a bit, without too much work.

See also Stepwise regression and Model selection on Wikipedia.

Answer 2

Stepwise regression basically fits the regression model by adding/dropping covariates one at a time based on a specified criterion (in your example above the criterion would be based on the BIC).

By specifying forward you are telling R that you would like to start with the simplest model (i.e., one covariate) and then add one covariate one at a time keeping only the ones that result in an improvement to the models BIC.

By specifying backward you are telling R that you want to start with the full model (i.e., the model with all the covariates) and then drop covariates, one ata time, that result in an improvement in the BIC.

Stepwise regression can be a very dangerous statistical procedure because it is not an optimal model selection procedure. The method can lead to very poor model selection because and it does not protect you against problems such as multiple comparisons.