# Metrics for selecting a logistic regression model

I have been running logistic regressions using forward, backward and 'both direction' stepwise procedures to guide the selection of the variables included in the model.

I have been using AIC as a metric for picking the better models. Initially I was reassured that this was a sensible approach in the post Model Selection: Logistic Regression

However I then found this post Algorithms for automatic model selection which seemed to suggest using AIC was not sensible (see the comments on the question by @gung which says AIC is not one of the better ways of selecting models.)

I was recently reading 'Introduction to Statistical Learning' by James, Witten, Hastie and Tibshirani where in the Lab on logistic regression (pp 159-160 in my printed copy) they create a hold out set and discuss the model quality based on the prediction accuracy on the hold out set. I could do that with my data too.

Q1. I'm confused that the 2 posts above seem to have differing views on using AIC to select models. Have I mis-interpreted them?

Q2. Is it possible to say the hold out set approach is superior to the AIC approach for selecting the logistic model?

• Please, please read the amazing amount of information on this site and elsewhere about the invalidity of all of the variable selection methods you are using. You are essentially just playing with data, avoiding statistical principles. May 28, 2018 at 11:37
• @FrankHarrell As should be evident from my question I have been reading the amazing amount of information on this site (referencing 2 pre-existing posts) and elsewhere ( referencing a text book).
– PM.
May 28, 2018 at 12:12
• @FrankHarrell What statistical principles are being avoided?
– PM.
May 28, 2018 at 12:27
• This is not even a complete list of the problems: stata.com/support/faqs/statistics/stepwise-regression-problems May 28, 2018 at 12:40
• @FrankHarrell Thanks, I have already seen that faq (it's referred to in the 2nd post I referenced above). From that I am happy to assume stepwise regression is a poor approach, despite it being described not unfavourably in textbooks (any wonder some of us get confused?). I'm still interested in answers to Q1 and Q2 tho'. Discussing stepwise regression doesn't answer Q1 or Q2 I don't think.
– PM.
May 28, 2018 at 12:53

I don't see my answers to the two questions as inconsistent with each other. There are a couple of basic ideas about feature selection that are two sides of the same coin:

1. Avoid selecting amongst a large number of options, and avoid using entirely empirical methods like stepwise selection routines (even based on the AIC).
2. Build models of substantive interest based on background knowledge.

I try to explain the problems with stepwise selection in my answer to Algorithms for automatic model selection. In the Model Selection: Logistic Regression thread, the OP describes a manual version of stepwise selection by selecting all the variables that are significant in univariate models and putting them into the final multiple regression model. The issue with that is that the problem with stepwise selection isn't primarily that the computer does it instead of you. The problems are intrinsic to selecting amongst large numbers of variables based on the same data for selecting and fitting / testing.

... evaluate models of substantive interest to you. Then use an information criterion that penalizes model flexibility (such as the AIC) to adjudicate amongst those models.

That is, you make only a few models (@FrankHarrell suggested you limit that to $$2$$ models) that are based on prior knowledge. Then, you can decide which seems to be better by the AIC. In this way, you are selecting amongst only a small number of options, each of which could be justified theoretically, and making only a single decision.

If you only care about out of sample predictive performance, it doesn't matter if you have the 'right' model, or if the variables in the model actually have true relationships with the outcome. It only matters if the model 'works' to give accurate predictions when you use it. The connection between the height of one twin and the height of their sibling is spurious / confounded, but who cares? You can predict the height of an unseen sibling with great accuracy and that's all you care about. In this type of situation, you can use cross validation and select the model with the best out of sample performance.

Please, please read the amazing amount of information on this site and elsewhere about the invalidity of all of the variable selection methods you are using. You are essentially just playing with data, avoiding statistical principles.

– Frank Harrell

( @FrankHarrell As should be evident from my question I have been reading the amazing amount of information on this site (referencing 2 pre-existing posts) and elsewhere (referencing a text book). @FrankHarrell What statistical principles are being avoided? – PM. )

This is not even a complete list of the problems: https://www.stata.com/support/faqs/statistics/stepwise-regression-problems/

– Frank Harrell

AIC is a simple translation of the p-value so doesn't solve any of the problems.

– Frank Harrell