I have been running logistic regressions using forward, backward and 'both direction' stepwise procedures to guide the selection of the variables included in the model.
I have been using AIC as a metric for picking the better models. Initially I was reassured that this was a sensible approach in the post Model Selection: Logistic Regression
However I then found this post Algorithms for automatic model selection which seemed to suggest using AIC was not sensible (see the comments on the question by @gung which says AIC is not one of the better ways of selecting models.)
I was recently reading 'Introduction to Statistical Learning' by James, Witten, Hastie and Tibshirani where in the Lab on logistic regression (pp 159-160 in my printed copy) they create a hold out set and discuss the model quality based on the prediction accuracy on the hold out set. I could do that with my data too.
Q1. I'm confused that the 2 posts above seem to have differing views on using AIC to select models. Have I mis-interpreted them?
Q2. Is it possible to say the hold out set approach is superior to the AIC approach for selecting the logistic model?