Better AIC but worse cross validation error rate

Question

I learn that AIC is usually used for assessing goodness of fit of a model and the criterion takes into account both goodness of fit and number of parameters used so that it could regulates the issue of overfitting.

While nowadays k-fold cross validation is commonly used for assessing model prediction performance. The two criterion should mostly align.

However recently I come across several instances in Kaggle where a model with better AIC result in worse cross validation error rate and test set error rate. While a model with worse AIC results in a better cross validated error rate and better test set error rate.

What are the reasons for such discrepency?

Answer 1

Sometimes the AIC can be more efficient because it makes use of the whole sample, and as such can select a larger number of predictors into the model. This is of course a good thing when you can be sure you are not overfitting the model. Many consider the penalty on AIC to be too lax. But that's not always the case.

On the other hand, cross-validation can be more stringent by validating the predictions in an external sample albeit many times. The variability of $k$-fold cross validation can be even more compromised when too few folds are performed. In settings where time is of the essence, the AICs rapid evaluation makes it possible to evaluate a space of models more quickly.

As others say in the comments, it all boils down to the data structure. With a very big $n$, the two approaches give nearly the same results. However, sparse data, $p \approx n$, and small sample sizes are scenarios when the choice of model selection strategy should be deliberate and well described. A sampling based validation strategy that some have advocated in small samples is the bootstrap since you don't exacerbate small sample size problems (I haven't had much luck with this approach).