2
$\begingroup$

I'm using AIC to compare a collection of submodels for a multivariate linear model. The best two models have very similar AIC scores. The model with the lowest AIC score includes all variables in the full model except one, and the second best model is the full model. The difference in score between the models is <2 (it's 0.3 in this case).

How should I interpret this result in deciding what model to use for subsequent analyses. Does it make more sense to use the simpler model because it is "just as good" as the full model and simpler, or to use the full model because it's not worse than the simpler model after model complexity is taken into account? Does it not matter at all? Or should I continue to investigate both models?

What's the rule here?

$\endgroup$
  • 2
    $\begingroup$ The textbook rule for AIC is to choose the model with the lowest AIC, period. Going beyond that requires using information about the specific models, how important the variable in question is, what the difference in likelihood translates to in interpretable units, what other models are under consideration, etc. $\endgroup$ – Kodiologist Sep 21 '17 at 21:11
  • 1
    $\begingroup$ I think it depends also on what you need to do with the models (e.g., test a hypothesis or predict new data?). If you need to predict new data it may be worth comparing the two models using cross-validation. $\endgroup$ – matteo Sep 21 '17 at 21:12
2
$\begingroup$

When there are several plausible models model uncertainty should not be ignored. If you select just one model and then do inference as if this were the only model you ever considered, you will get problems (estimates, p-values, confidence intervals etc. will not have their usual repeat sampling properties, out of sample predictions will be problematic etc.).

It is usually preferable to account for model uncertainty by e.g. doing model averaging (e.g. with weights proportional to exp(-(AIC-min AIC)/2)) unless one model is so much better in terms of AIC that it gets all the weight anyway.

$\endgroup$
4
$\begingroup$

How should I interpret this result in deciding what model to use for subsequent analyses.

Essentially the AIC offers a way to compare how much an increase in a model's likelihood is "worth" an increase in the model's complexity when determining model quality. Since you propose two models with similar AIC's what you should interpret is that according to this measure of likelihood vs. complexity trade-off they are essentially equal in quality.

Does it make more sense to use the simpler model because it is "just as good" as the full model and simpler

Not really as it is only "just as good" when simplicity has already been taken into account.

or to use the full model because it's not worse than the simpler model after model complexity is taken into account?

Again same answer but in reverse. The gain in simplicity is approximately balanced out by the difference in likelihood.

Does it not matter at all?

Yes and no. They are two different models and from a frequentist point of view there is only one "correct" model which describes the data, which could even in fact (and probably more likely) be neither of these. So in this sense it does matter which model you choose. But determining this through AIC shows choosing one or the other does not change the quality by any meaningful amount. Essentially do you prefer a model with greater likelihood to fit the data but with more complexity or vice versa, where the trade-off according to Akaike is essentially equal.

Or should I continue to investigate both models?

Do you expect to gain different but still meaningful insights through both models? Refer to @Bjorn

$\endgroup$
  • 3
    $\begingroup$ +1, but in the last point I think model averaging is an attractive alternative when AIC values are similar (as @Björn mentions in his answer). Regarding from a frequentist point of view there is a "correct" model to choose, it need not be one of the two models being considered. $\endgroup$ – Richard Hardy Sep 22 '17 at 5:19
  • $\begingroup$ @Richard Hardy good points. I have edited the answer where it was unclear $\endgroup$ – Dale C Sep 22 '17 at 7:44
0
$\begingroup$

Without more information, select the model with lowest AIC. But in most scenarios, I'd be comparing each model's RMSE or MAE as well - attempting to maximize prediction accuracy on the test set.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.