I've taken a regression class and am now in a machine learning class. In regression, we talk about model selection using adj-R2 and AIC/BIC. In my machine learning class, we primarily select models using cross validation (or sample splitting) and the validation error.

I am not seeing the connection between the two, or more specifically the divide between the two. Seems like cross validation/sample splitting can work on any model, so why even bother with adj-R2 or AIC/BIC?

When would we want to use one over the other? And are there situations where AIC/BIC wouldn't work?



The question is quite broad, but I will give some starting points:

Why bother with AIC/BIC: using cross validation (CV) is (much) more computationally expensive than using AIC/BIC, except for some special cases like leave-one-out cross validation (LOOCV) for regression where it is computationally as cheap as AIC/BIC.

Situations where AIC/BIC would not work: AIC/BIC are only available for models estimated using maximum likelihood estimation (MLE), and this is a relative small class of models in the context of machine learning.

Connection between CV and AIC/BIC: under some assumptions, AIC is asymptotically equivalent to LOOCV while BIC is asymptotically equivalent to k-fold CV with a specific fold size that depends on the sample size. So under these assumptions, you can save a lot of computations by replacing CV with AIC/BIC.

On $R^2_{adj.}$: According to "Justification for and optimality of $R^2_{adj.}$ as a model selection criterion", it is questionable whether $R^2_{adj.}$ can be regarded as an optimal model selection criterion. Personally, I would not use it when other alternatives like AIC, BIC or CV are available.

| cite | improve this answer | |
  • $\begingroup$ LOOCV is computationally cheap? How is that not way more expensive than 5-fold CV? If I have 1000 observations, I have to fit 1000 regression models to run LOOCV. $\endgroup$ – Dave Feb 9 at 12:49
  • $\begingroup$ @Dave, no, there is a neat workaround that does LOOCV for regression in just one fit, using the hat matrix (I think; or maybe another one) to obtain the errors for each of the $n$ folds. This only works for regression, though. There might be another thread describing it, otherwise you should be able to find it in a textbook. $\endgroup$ – Richard Hardy Feb 9 at 12:55
  • $\begingroup$ Ah thanks, the second one is a good reason that I am not aware about. As my classes aren't taught by the same professor, there is no flow. Just suddenly we switched from AIC to CV. I would imagine AIC might be more useful in situations where we have time series data. $\endgroup$ – confused Feb 10 at 18:32
  • $\begingroup$ @confused, yes, in a time series setting, CV becomes conceptually more complicated, while AIC/BIC are still as simple as they are otherwise. $\endgroup$ – Richard Hardy Feb 11 at 6:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.