2
$\begingroup$

I have been wondering, is there any case in which AIC should be avoided as the evaluation metric? I cannot really find anything but for the advantages - what about the disadvantages? I'm mostly interested in time series models - assuming it changes anything.

$\endgroup$
2

1 Answer 1

3
$\begingroup$

AIC is asymptotically identical to leave-one-out cross-validation. Thus, you should use it any time you would use CV to select your model, which is mainly when you want to minimise predictive error.

The disadvantage is that in a limited data situation, AIC will not select the causal model, and in the large-data limit, AIC will select more complicated models than BIC, and is not necessarily asymptotically consistent if the true model is in the set.

I would say the bottomline is that AIC is a good criteria if you want to minimise predictive error in a data-limited situation.

$\endgroup$
7
  • $\begingroup$ Wouldn't AIC lead to overfitting in data-limited situation? $\endgroup$
    – Fatafim
    Oct 7, 2022 at 13:29
  • $\begingroup$ Why would you think so? I mean, for small data, you can use AICc, but in principle, I would say AIC is a metric that explicitly aims at avoiding overfitting in model predictions $\endgroup$ Oct 8, 2022 at 16:40
  • $\begingroup$ Because AIC - as far as I know but I might be wrong here - tends to choose more complex models than BIC, which is what I find prone to overfitting $\endgroup$
    – Fatafim
    Oct 8, 2022 at 17:43
  • $\begingroup$ Yes, but differences between AIC and BIC get larger in the large-data limit, so this is in the large-data limit $\endgroup$ Oct 9, 2022 at 7:32
  • 1
    $\begingroup$ BIC penalizes k log(n) and AIC with 2k, so AIC = BIC for data size n = exp(2) ~ 7. $\endgroup$ Oct 10, 2022 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.