# Selecting the model by bootstrapping: AIC vs. log-loss?

I'm building a predictive model with potentially multiple predictors. To that end, I try different, nested models, each with one more predictor than the previous one and compare their AICs. The AIC falls with each new predictor, but very slowly after the second one. Since the AIC is itself a random variable, I worry that a formally better model, where the AIC is lower by less than 0.5% than the previous one, is not truly better, but just a random effect.

So I thought I'd compare the models by bootstrapping. There are at least two ways I can think of:

1. For each set of predictors, generate 1000 (or whatever) different bootstrap datasets, fit a model on each dataset and record its AIC. Plot the distribution of AICs over different set of predictors ('Full model' corresponds to 'AWFST' in the boxplot):

Or:

1. For each set of predictors, train the model on the full dataset. Generate 1000 different bootstrap datasets, use each model to make predictions on each dataset and record its log-loss. Plot the distribution of log-losses over different set of predictors:

For better comparison, the same random seed was used in both approaches. As you can see, the results are quite similar, but not quite identical. Does any of the approaches make sense and, if yes, is one 'better' than the other? If not, where am I making a mistake?

• Are you testing in-sample or on some holdout data?
– Dave
Commented Mar 22, 2023 at 17:46
• @Dave In-sample. Commented Mar 22, 2023 at 18:51
• Then how are you penalizing the model for overfitting when it comes to log-loss?
– Dave
Commented Mar 22, 2023 at 18:52
• While bootstrapping tells you something about the variability in AIC, the vanilla AIC is still your best estimate (of twice the negative expected likelihood of a new observation), and therefore it makes sense to use vanilla AIC in model selection (unless you prefer worse estimates to the best estimate). Commented Mar 23, 2023 at 8:09
• I am with Richard on this (+1). Avoid inventing a wheel, that looks like an existing wheel but it is not the existing wheel. (The wheel here being AIC) Commented Mar 23, 2023 at 11:24

Regarding bootstrapping, it tells you something about the variability in AIC. However, the vanilla AIC is still your best estimate (of $$-2n\ \times$$ the expected log-likelihood of a new observation), and therefore it makes sense to use vanilla AIC in model selection – unless you prefer worse estimates to the best estimate...