4
$\begingroup$

I learn that AIC is usually used for assessing goodness of fit of a model and the criterion takes into account both goodness of fit and number of parameters used so that it could regulates the issue of overfitting.

While nowadays k-fold cross validation is commonly used for assessing model prediction performance. The two criterion should mostly align.

However recently I come across several instances in Kaggle where a model with better AIC result in worse cross validation error rate and test set error rate. While a model with worse AIC results in a better cross validated error rate and better test set error rate.

What are the reasons for such discrepency?

$\endgroup$
  • $\begingroup$ Just a minor comment: Minimizing the AIC is asymptotically equivalent to minimizing the CV value, see Stone M (1977). See Rob Hyndman's comments on cross-validation for more information. $\endgroup$ – COOLSerdash Nov 29 '18 at 14:41
  • 1
    $\begingroup$ @COOLSerdash, specifically, AIC ~ LOOCV while BIC ~ a certain form of K-fold CV. So no wonder if AIC differs from K-fold CV that is not LOOCV. $\endgroup$ – Richard Hardy Nov 29 '18 at 15:12
  • $\begingroup$ @RichardHardy that is an interesting result. Can you point me to where that is proved? Interested in the methods used. $\endgroup$ – AdamO Nov 30 '18 at 15:31
  • $\begingroup$ I discovered this somewhere on CV. Regarding AIC, the proof is probably in the linked paper. Regarding BIC, I do not remember, sorry. But I am sure some of the threads includes a reference. $\endgroup$ – Richard Hardy Nov 30 '18 at 16:33
2
$\begingroup$

Sometimes the AIC can be more efficient because it makes use of the whole sample, and as such can select a larger number of predictors into the model. This is of course a good thing when you can be sure you are not overfitting the model. Many consider the penalty on AIC to be too lax. But that's not always the case.

On the other hand, cross-validation can be more stringent by validating the predictions in an external sample albeit many times. The variability of $k$-fold cross validation can be even more compromised when too few folds are performed. In settings where time is of the essence, the AICs rapid evaluation makes it possible to evaluate a space of models more quickly.

As others say in the comments, it all boils down to the data structure. With a very big $n$, the two approaches give nearly the same results. However, sparse data, $p \approx n$, and small sample sizes are scenarios when the choice of model selection strategy should be deliberate and well described. A sampling based validation strategy that some have advocated in small samples is the bootstrap since you don't exacerbate small sample size problems (I haven't had much luck with this approach).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.