Choice of models with nearly equal AIC values?

Question

I'm using AIC to compare a collection of submodels for a multivariate linear model. The best two models have very similar AIC scores. The model with the lowest AIC score includes all variables in the full model except one, and the second best model is the full model. The difference in score between the models is <2 (it's 0.3 in this case).

How should I interpret this result in deciding what model to use for subsequent analyses. Does it make more sense to use the simpler model because it is "just as good" as the full model and simpler, or to use the full model because it's not worse than the simpler model after model complexity is taken into account? Does it not matter at all? Or should I continue to investigate both models?

What's the rule here?

Answer 1

When there are several plausible models model uncertainty should not be ignored. If you select just one model and then do inference as if this were the only model you ever considered, you will get problems (estimates, p-values, confidence intervals etc. will not have their usual repeat sampling properties, out of sample predictions will be problematic etc.).

It is usually preferable to account for model uncertainty by e.g. doing model averaging (e.g. with weights proportional to exp(-(AIC-min AIC)/2)) unless one model is so much better in terms of AIC that it gets all the weight anyway.

Answer 2

How should I interpret this result in deciding what model to use for subsequent analyses.

Essentially the AIC offers a way to compare how much an increase in a model's likelihood is "worth" an increase in the model's complexity when determining model quality. Since you propose two models with similar AIC's what you should interpret is that according to this measure of likelihood vs. complexity trade-off they are essentially equal in quality.

Does it make more sense to use the simpler model because it is "just as good" as the full model and simpler

Not really as it is only "just as good" when simplicity has already been taken into account.

or to use the full model because it's not worse than the simpler model after model complexity is taken into account?

Again same answer but in reverse. The gain in simplicity is approximately balanced out by the difference in likelihood.

Does it not matter at all?

Yes and no. They are two different models and from a frequentist point of view there is only one "correct" model which describes the data, which could even in fact (and probably more likely) be neither of these. So in this sense it does matter which model you choose. But determining this through AIC shows choosing one or the other does not change the quality by any meaningful amount. Essentially do you prefer a model with greater likelihood to fit the data but with more complexity or vice versa, where the trade-off according to Akaike is essentially equal.

Or should I continue to investigate both models?

Do you expect to gain different but still meaningful insights through both models? Refer to @Bjorn

Answer 3

Without more information, select the model with lowest AIC. But in most scenarios, I'd be comparing each model's RMSE or MAE as well - attempting to maximize prediction accuracy on the test set.