I have read many posts (including Frank Harrell's book) about the consequences of using variable selection strategies.

However, it seems that many of the published work in the medical field still follow step-wise, forward and backward selection methods.

variable selection using backward elimination cites Frank Harrell's book and have done a model selection based on AIC and then started interpreting the coefficients.

Is it meaningful to do in practice, in any sense? If they are not legitimate, why people keep using it and getting published in journals?

  • $\begingroup$ Why is it not legitimate? It is, but you should understand the inner works. Reason #1: step-wise methods are greedy and that means you will probably get settled with suboptimal set of predictors in some way. You want search CrossValidated with this keyword to get knowledge. Reason #2: AIC is biased, which means you may end up with abundant set of predictors, or a kind of set that will show strange p-values for some coefficients. There is also the BIC criterion which is somewhat more robust. And again you want to search through some literature to understand how they work inside. $\endgroup$ – Alexey Burnakov Oct 10 '17 at 8:29
  • $\begingroup$ Sorry, I don't get your point. If the variable selection strategies work poorly in practice (as you have also mentioned), why people interpret unreliable coefficient estimates and their p-values? $\endgroup$ – shani Oct 10 '17 at 8:42
  • $\begingroup$ Let me add that when people use any feature selection method and build a model involving the coefficients, like a linear regression or logistic regression model, they study the resulted model's coefficients first and after that proceed to interpretation. If the feature selection provided poor results, then this model is not ready for article publishing. Often a researcher tries several alternatives before they get satisfied with the results. Often a method that minimizes the AIC provides a researcher with a satisfactory model, albeit suboptimal. $\endgroup$ – Alexey Burnakov Oct 10 '17 at 8:47
  • $\begingroup$ One more aspect to this question which is very practical. Any greedy method is much less expensive compared to any stochastic (non-greedy) approach. It works fast, in other words. You can run a step-wise selection for the logistic model in 10 lines of code and in a fraction of minute you will have the selected model. Another important fact is that AIC based model selection has a long history of application in the econometrics community. So it has a wide support compared to some other methods. Details of application depend on your data, and a model you prefer to fit. $\endgroup$ – Alexey Burnakov Oct 10 '17 at 9:25
  • 2
    $\begingroup$ The fact that so many papers are using stepwise regression doesn't mean that it's OK. The vast majority of researchers are unschooled in the horrendous performance of stepwise methods, and don't understand how selection bias hurts regression coefficients and even worse makes standard errors much too small. And if you don't follow that, just know that stepwise regression has almost zero probability of selecting the "right" variables. $\endgroup$ – Frank Harrell Oct 10 '17 at 12:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.