15
$\begingroup$

I understand AIC is asymptotically equivalent to leave-one-out cross-validation and that BIC has a similar asymptotic equivalence to leave-k-out cross-validation. My question is, other than computational efficiency, is there a reason to prefer information criteria like AIC/wAIC/BIC over cross-validation?

In the ecology literature, using AIC for model selection is very common. But if cross-validation is possible, is there a reason to use AIC?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Welcome to the site, i am interested in others' responses. Dear CV community: It seems this questions was asked before but not answered <stats.stackexchange.com/questions/201568/…>. I will give one reason in an answer on this question (unless it would be better for me to answer the old question). $\endgroup$
    – John Madden
    Commented Nov 24 at 17:56
  • 1
    $\begingroup$ Related: AIC versus cross validation in time series: the small sample case. $\endgroup$
    – Richard Hardy
    Commented Nov 24 at 18:56
  • 5
    $\begingroup$ A downside of information criteria is they depend on sample size (N) and thus are not comparable between models fit with different N. In contrast, many CV-based estimates (e.g. mean absolute error for numeric outcomes or AUC for a categorical outcome) can be compared between different models, say from different publications, that have different sample sizes, as long as they are studying the same outcome. Another downside of information criteria is they only apply to likelihood-based models, wheras CV can be used for ANY model, such as tree-based methods that do not use likelihood. $\endgroup$
    – jarbet
    Commented Nov 25 at 23:29

2 Answers 2

Reset to default
14
$\begingroup$

Not a comprehensive answer, but merely a reason I often run into: correlated errors. Fundamentally, the problem in using CV with correlated errors is that the train and test sets contain information about one another.

Cross-Validation is most straightforward when we have exchangeable data, e.g. images of cats. But if we have data with, say, spatiotemporal error structure, it becomes less satisfying. Sure, we can use Blocked Cross Validation, which holds out groups of highly correlated data instead of randomly sampling them. But personally this just doesn't feel quite as clean as CV in the exchangeable case.

In such situations, I think alternative approaches like information criteria or Bayesian model selection become more attractive.

Improve this answer
$\endgroup$
4
  • 2
    $\begingroup$ Aki Vehtari's excellent FAQ on cross-validation notes that at least in the time series case, leave-one-out cross validation is often preferable to leave-future-out cross validation. The correlated errors induce bias, but this bias tends to be small empirically and you get much lower variance of the estimates then when leaving out larger blocks. users.aalto.fi/~ave/… Additionally, the bias has to translate to IC, as those are an approximation to leave-one-out CV. $\endgroup$
    – Martin Modrák
    Commented Nov 25 at 13:02
  • $\begingroup$ Correlated errors is an interesting point. However, its not obvious to me that AIC is not also affected by these to the same degree. For example, imagine you're using AIC or LOOCV for model selection. Its obvious that in a single cross-validation instance the correlated errors are a problem. But once you iterate over every datapoint, how is LOOCV going to in some sense be 'worse' than AIC? $\endgroup$
    – Louis F-H
    Commented Dec 5 at 18:06
  • 1
    $\begingroup$ @LouisF-H AIC is based on the likelihood, and you can just bake the correlation into that. For instance, in spatial stats, people do AIC with a Gaussian process error distribution. The problem for CV does not go away if you average over many points. For each iteration of LOOCV, the point left out has "more similarity" with the in-sample points than it would in the iid case, which means the error is biased low. Then averaging over iterations reduces variance but does not affect the bias. $\endgroup$
    – John Madden
    Commented Dec 5 at 18:10
  • 1
    $\begingroup$ Thanks for the explanation. I hadn't appreciated that its possible to account for correlated errors in the likelihood. A GP error is a helpful example. $\endgroup$
    – Louis F-H
    Commented Dec 5 at 20:11
13
$\begingroup$

To add to the previous answers, information criteria might sometimes be more practical than cross-validation:

  • An information criterion computation requires only one optimization (e.g. the argmax of $p(\mathbf{D}|\theta)$ for the BIC), while you have to perform data fitting for each fold in CV. This can be time-consuming if fitting is computationally challenging (for instance if your data are correlated and optimization needs to be performed using the EM algorithm).
  • If only few samples are available to fit a model, it might not be possible to train it on only a subset of the data. Information criteria do not require to split between a training and a testing set.
Improve this answer
$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.