I have been running logistic regressions using forward, backward and 'both direction' stepwise procedures to guide the selection of the variables included in the model.

I have been using AIC as a metric for picking the better models. Initially I was reassured that this was a sensible approach in the post Model Selection: Logistic Regression

However I then found this post Algorithms for automatic model selection which seemed to suggest using AIC was not sensible (see the comments on the question by @gung which says AIC is not one of the better ways of selecting models.)

I was recently reading 'Introduction to Statistical Learning' by James, Witten, Hastie and Tibshirani where in the Lab on logistic regression (pp 159-160 in my printed copy) they create a hold out set and discuss the model quality based on the prediction accuracy on the hold out set. I could do that with my data too.

Q1. I'm confused that the 2 posts above seem to have differing views on using AIC to select models. Have I mis-interpreted them?

Q2. Is it possible to say the hold out set approach is superior to the AIC approach for selecting the logistic model?

  • $\begingroup$ Please, please read the amazing amount of information on this site and elsewhere about the invalidity of all of the variable selection methods you are using. You are essentially just playing with data, avoiding statistical principles. $\endgroup$ – Frank Harrell May 28 '18 at 11:37
  • $\begingroup$ @FrankHarrell As should be evident from my question I have been reading the amazing amount of information on this site (referencing 2 pre-existing posts) and elsewhere ( referencing a text book). $\endgroup$ – PM. May 28 '18 at 12:12
  • $\begingroup$ @FrankHarrell What statistical principles are being avoided? $\endgroup$ – PM. May 28 '18 at 12:27
  • $\begingroup$ This is not even a complete list of the problems: stata.com/support/faqs/statistics/stepwise-regression-problems $\endgroup$ – Frank Harrell May 28 '18 at 12:40
  • $\begingroup$ @FrankHarrell Thanks, I have already seen that faq (it's referred to in the 2nd post I referenced above). From that I am happy to assume stepwise regression is a poor approach, despite it being described not unfavourably in textbooks (any wonder some of us get confused?). I'm still interested in answers to Q1 and Q2 tho'. Discussing stepwise regression doesn't answer Q1 or Q2 I don't think. $\endgroup$ – PM. May 28 '18 at 12:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.