4
$\begingroup$

I'm building a predictive model with potentially multiple predictors. To that end, I try different, nested models, each with one more predictor than the previous one and compare their AICs. The AIC falls with each new predictor, but very slowly after the second one. Since the AIC is itself a random variable, I worry that a formally better model, where the AIC is lower by less than 0.5% than the previous one, is not truly better, but just a random effect.

So I thought I'd compare the models by bootstrapping. There are at least two ways I can think of:

  1. For each set of predictors, generate 1000 (or whatever) different bootstrap datasets, fit a model on each dataset and record its AIC. Plot the distribution of AICs over different set of predictors ('Full model' corresponds to 'AWFST' in the boxplot):

AIC bootstrap (scatterplot) AIC bootstrap (boxplot)

Or:

  1. For each set of predictors, train the model on the full dataset. Generate 1000 different bootstrap datasets, use each model to make predictions on each dataset and record its log-loss. Plot the distribution of log-losses over different set of predictors:

Log-loss bootstrap (scatterplot) Log-loss bootstrap (boxplot)

For better comparison, the same random seed was used in both approaches. As you can see, the results are quite similar, but not quite identical. Does any of the approaches make sense and, if yes, is one 'better' than the other? If not, where am I making a mistake?

$\endgroup$
14
  • $\begingroup$ Are you testing in-sample or on some holdout data? $\endgroup$
    – Dave
    Mar 22, 2023 at 17:46
  • $\begingroup$ @Dave In-sample. $\endgroup$
    – Igor F.
    Mar 22, 2023 at 18:51
  • $\begingroup$ Then how are you penalizing the model for overfitting when it comes to log-loss? $\endgroup$
    – Dave
    Mar 22, 2023 at 18:52
  • 1
    $\begingroup$ While bootstrapping tells you something about the variability in AIC, the vanilla AIC is still your best estimate (of twice the negative expected likelihood of a new observation), and therefore it makes sense to use vanilla AIC in model selection (unless you prefer worse estimates to the best estimate). $\endgroup$ Mar 23, 2023 at 8:09
  • 1
    $\begingroup$ I am with Richard on this (+1). Avoid inventing a wheel, that looks like an existing wheel but it is not the existing wheel. (The wheel here being AIC) $\endgroup$
    – usεr11852
    Mar 23, 2023 at 11:24

1 Answer 1

3
$\begingroup$

AIC is optimal in a well defined sense under a certain set of assumptions. You seem to be looking for a reason not to use AIC. If so, might it be that the goal of your analysis does not justify AIC as a model selection criterion to begin with? Or might it be that it does, in which case you might be straying away from the optimal choice without a theoretically sound argument?

I worry that a formally better model, where the AIC is lower by less than 0.5% than the previous one, is not truly better, but just a random effect.

But are you not worried about the opposite effect which is probably equally likely: that save for a random effect, drops are larger than 0.5%? If you think you can improve upon AIC, use the improved criterion. This criterion, however, suggests what it does, and I do not see a way around that. I am afraid you might fool yourself by using ad-hoc adjustments.

Regarding bootstrapping, it tells you something about the variability in AIC. However, the vanilla AIC is still your best estimate (of $-2n\ \times$ the expected log-likelihood of a new observation), and therefore it makes sense to use vanilla AIC in model selection – unless you prefer worse estimates to the best estimate...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.