I have been wondering, is there any case in which AIC should be avoided as the evaluation metric? I cannot really find anything but for the advantages - what about the disadvantages? I'm mostly interested in time series models - assuming it changes anything.
$\begingroup$
$\endgroup$
2
-
4$\begingroup$ E.g., when you are confident that the true model is in the set of the models considered, there are good reasons to prefer the BIC, see e.g. stats.stackexchange.com/questions/197112/… $\endgroup$– Christoph HanckCommented Oct 6, 2022 at 12:18
-
2$\begingroup$ The AIC also performs better than the BIC in the tapering effect context, see stats.stackexchange.com/a/493214/271601 $\endgroup$– Camille GontierCommented Oct 6, 2022 at 13:01
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
7
AIC is asymptotically identical to leave-one-out cross-validation. Thus, you should use it any time you would use CV to select your model, which is mainly when you want to minimise predictive error.
The disadvantage is that in a limited data situation, AIC will not select the causal model, and in the large-data limit, AIC will select more complicated models than BIC, and is not necessarily asymptotically consistent if the true model is in the set.
I would say the bottomline is that AIC is a good criteria if you want to minimise predictive error in a data-limited situation.
-
$\begingroup$ Wouldn't AIC lead to overfitting in data-limited situation? $\endgroup$– FatafimCommented Oct 7, 2022 at 13:29
-
$\begingroup$ Why would you think so? I mean, for small data, you can use AICc, but in principle, I would say AIC is a metric that explicitly aims at avoiding overfitting in model predictions $\endgroup$ Commented Oct 8, 2022 at 16:40
-
$\begingroup$ Because AIC - as far as I know but I might be wrong here - tends to choose more complex models than BIC, which is what I find prone to overfitting $\endgroup$– FatafimCommented Oct 8, 2022 at 17:43
-
$\begingroup$ Yes, but differences between AIC and BIC get larger in the large-data limit, so this is in the large-data limit $\endgroup$ Commented Oct 9, 2022 at 7:32
-
1$\begingroup$ BIC penalizes k log(n) and AIC with 2k, so AIC = BIC for data size n = exp(2) ~ 7. $\endgroup$ Commented Oct 10, 2022 at 14:49