Clarification of the stepwise regression analysis : Bidirectional elimination

Question

I am working on a large dataset (2000 patients, 100 variable) My purpose is to identify the influence of the variable (VarX1, VarX2, VarX3,…..) on a dichotomial outcome (VarY).

My purpose is to perform a manual Stepwise regression analysis on a logistic regression in order to select the best variable to explain the outcome (VarY). I have several problems understanding the stepwise regression analysis; here is what I understand so far:

1. I am starting with the null model. : VarY~1

2. I add all variables consecutively into the model (VarY~VarX1, VarY~VarX2, VarY~VarX3….) and select the Var with the minimum p value and a p value above 0.05 Here is my first question: For a quantitative variable with 3 parameters (example: color: blue/yellow/green), there will be two beta, and two p value estimated. How can I deal with that, especially if there is one pvalue >0.05 and another one <0.05 ?

3. I add the variable with the least p value into my model. Let’s say that VarX5 is added into the model
4. I repeat number 2 in order to add one variable. Let’s say I add VarX8 into the model : VarY~VarX5+VarX8
5. I am supposed to make a backward selection, and this is beginning to be “fuzzy” : I should retrieve only the higher p value? When I have more variables, and at one point, two or three variable have a p value over 0.05, how should I manage that? Or I should retrieve only the last variable added into the model (for example, for the model into step 4, I retrieve only VarX5?

Thank you for your help, or for any of the article/tutorial/link you can give me.

EDIT : Thanks to those who gave me some answers. In order to clarify, I am aware of the limitation of the stepwise analysis. My purpose is to understand how it works. And if several publication point out the failure of this analysis, I could not find a clear one that explain all the processes.

EDIT 2 : I have seen the method with the AIC and the BIC, which are simpler and answers both questions. ;) This is the method with the p value that I am concerned about.

Answer 1

You should not do this at all. Stepwise regression yields output that is all incorrect:

• Parameter estimates are biased away from 0
• p values are too small
• Standard errors are too small

Worst of all, it allows the analyst not to think. In fact, truly automatizing the model building process prevents the analyst from thinking.

It is a way to tell your boss that you should make less money.

Answer 2

For single predictors, the AIC and p-value criteria are effectively the same, with the AIC criterion equivalent to a p-value cutoff of 0.157. That's a very strict criterion to use in real-world applications with correlations among predictors and when, by design, a stepwise approach that starts in the forward direction doesn't take into account the associations of the omitted predictors with outcome. A forward approach using such a strict cutoff might miss important real relationships that would be found with a less strict cutoff. So the problem with forward stepwise isn't just that it can find associations with outcome that aren't really there. Your proposed use of p < 0.05 in your question is almost certainly much too strict and could easily lead to false-negative conclusions.

As you are doing logistic regression the problem with the forward approach is exacerbated. Unlike linear regression, omitting any predictor associated with outcome leads to downward bias in the magnitudes of the coefficients of the included predictors, even if the omitted predictors are uncorrelated with the included ones. So there's a serious risk that the p values will be higher than they would be if you had been using a model including multiple predictors to start.

The questions you raise with respect to categorical predictors having more than 2 levels might be one reason why you see so little stepwise use of p-value criteria. What would usually make logical sense would be to rule the entire predictor, with all of its levels, in or out of the model. The AIC criterion does that quite naturally. The p values returned for such a predictor in standard summaries are for the differences of the coefficient for each level from the reference level, so you have multiple p values that depend on the choice of the reference level. What is then needed would be a test like an ANOVA F-test to get a p value that evaluates all levels of the predictor. Even if you did that, however, as I understand this answer the statistic wouldn't properly follow the F distribution so the reported p values would be meaningless.

Other than that, your understanding of the initially forward but bidirectional stepwise approach, allowing for removal as well as addition of predictors, is pretty much as it is implemented in functions like the step() function in R, which uses subfunctions add1() and drop1() to add/subtract predictors individually at each step, where "individually" means entire categorical predictors with all of their levels. Looking at the open-source code for those functions might be helpful.

If for some reason you need to do predictor selection and wish to understand how it is implemented, you will be better off learning about LASSO, which has a more principled basis and penalizes regression coefficients to minimize the overfitting that is endemic with standard stepwise approaches. The issue you raise about multi-level categorical predictors is then handled by the group LASSO.