I have read many posts (including Frank Harrell's book) about the consequences of using variable selection strategies.

However, it seems that many of the published work in the medical field still follow step-wise, forward and backward selection methods.

variable selection using backward elimination cites Frank Harrell's book and have done a model selection based on AIC and then started interpreting the coefficients.

Is it meaningful to do in practice, in any sense? If they are not legitimate, why people keep using it and getting published in journals?

  • $\begingroup$ Why is it not legitimate? It is, but you should understand the inner works. Reason #1: step-wise methods are greedy and that means you will probably get settled with suboptimal set of predictors in some way. You want search CrossValidated with this keyword to get knowledge. Reason #2: AIC is biased, which means you may end up with abundant set of predictors, or a kind of set that will show strange p-values for some coefficients. There is also the BIC criterion which is somewhat more robust. And again you want to search through some literature to understand how they work inside. $\endgroup$ Oct 10, 2017 at 8:29
  • $\begingroup$ Sorry, I don't get your point. If the variable selection strategies work poorly in practice (as you have also mentioned), why people interpret unreliable coefficient estimates and their p-values? $\endgroup$
    – shani
    Oct 10, 2017 at 8:42
  • $\begingroup$ Let me add that when people use any feature selection method and build a model involving the coefficients, like a linear regression or logistic regression model, they study the resulted model's coefficients first and after that proceed to interpretation. If the feature selection provided poor results, then this model is not ready for article publishing. Often a researcher tries several alternatives before they get satisfied with the results. Often a method that minimizes the AIC provides a researcher with a satisfactory model, albeit suboptimal. $\endgroup$ Oct 10, 2017 at 8:47
  • 3
    $\begingroup$ The fact that so many papers are using stepwise regression doesn't mean that it's OK. The vast majority of researchers are unschooled in the horrendous performance of stepwise methods, and don't understand how selection bias hurts regression coefficients and even worse makes standard errors much too small. And if you don't follow that, just know that stepwise regression has almost zero probability of selecting the "right" variables. $\endgroup$ Oct 10, 2017 at 12:19
  • 2
    $\begingroup$ Stepwise selection using AIC or any other method that uses just the same information that p-values uses is highly problematic. You should search this site for the many, many posts on the subject. See also stata.com/support/faqs/statistics/stepwise-regression-problems $\endgroup$ Oct 16, 2017 at 13:21


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.