# What is the point of dividing data into training and test parts to assess prediction properties when we have AIC?

Asymptotically, minimizing the AIC is equivalent to minimizing the leave-one-out cross-validation MSE for cross-sectional data [1]. So when we have AIC, why does one at all use the method of dividing the data into training, validation and test sets to measure the predictive properties of models? What specifically are the benefits of this practice?

I can think of one reason: if one wants to assess the models' predictive performances, out-of-sample analysis is useful. But although AIC is not a measure of forecast accuracy, one usually has a good idea if some model is reaching its maximum potential (for the data one is given) in terms of how well you are gonna be able to predict.

• An excerpt from sklearn's docs: Information-criterion based model selection is very fast, but it relies on a proper estimation of degrees of freedom, are derived for large samples (asymptotic results) and assume the model is correct, i.e. that the data are actually generated by this model. They also tend to break when the problem is badly conditioned (more features than samples). – sascha May 27 '16 at 11:48
• I do not actually think that AIC assumes a correct model (stats.stackexchange.com/questions/205222/…). Regarding sample size and AIC being an asymptotic result: you would never divide your data into three parts when you have little data. So small sample size is problematic for both out-of-sample analysis and AIC – Erosennin May 27 '16 at 11:58
• @sascha has a point there: for AIC to approximate expected KL info. loss well one of the models has to be fairly good. I don't think anyone advocates using AIC to compare bad models to see which is less bad. – Scortchi May 29 '16 at 18:09
• $\operatorname{tr}(J(\theta_0)(I(\theta_0))^{-1}) \approx k$ in slide 10 that @sascha linked to. (I was just looking on our site - we seem to have a lot of assertions about AIC, & references containing yet more assertions; but little beyond. From memory, Pawitan, In All Likelihood, & Burnham & Anderson, Model Selection, give derivations.) – Scortchi May 29 '16 at 18:22
• Ok, I skipped the TIC-part and missed that bit. You are absolutely right. Apologies to you @sascha , and thank you for enlightening me :) Yes, I just had a look in Burnham & Anderson myself. Great resource! – Erosennin May 29 '16 at 18:41

• In respect of 2. there's a relatively easy way to get the degrees of freedom of the model (though in some cases it may be moderately time consuming to compute, in many common situations there's a shortcut); which is $k=\sum_i \frac{\partial \hat{y}_i}{\partial y_i}$; in a quite literal direct sense this measures the model's degrees of freedom to approximate the data. See for example Ye's 1998 JASA article. StasK links to a full reference in this answer for example. ... ctd – Glen_b Jun 7 '16 at 0:34