What is the point of dividing data into training and test parts to assess prediction properties when we have AIC? Asymptotically, minimizing the AIC is equivalent to minimizing the leave-one-out cross-validation MSE for cross-sectional data [1]. So when we have AIC, why does one at all use the method of dividing the data into training, validation and test sets to measure the predictive properties of models? What specifically are the benefits of this practice?
I can think of one reason: if one wants to assess the models' predictive performances, out-of-sample analysis is useful. But although AIC is not a measure of forecast accuracy, one usually has a good idea if some model is reaching its maximum potential (for the data one is given) in terms of how well you are gonna be able to predict.
 A: In practice, I always use cross-validation or a simple train–test split rather than AIC (or BIC). I'm not too familiar with the theory behind AIC, but two chief concerns lead me to prefer more direct estimates of predictive accuracy:


*

*The number itself doesn't tell you much about how accurate a model is. AIC can provide evidence as to which of several models is the most accurate, but it doesn't tell you how accurate the model is in units of the DV. I'm almost always interested in concrete accuracy estimates of this kind, because it tells me how useful a model is in absolute terms, and also how much more accurate it is than a comparison model.

*AIC, like BIC, needs for each model a parameter count or some other value that measures the model's complexity. It isn't clear what you should do for this in the case of less traditional predictive methods like nearest-neighbor classification, random forests, or the wacky new ensemble method you scribbled onto a cocktail napkin midway through last month's bender. By contrast, accuracy estimates can be produced for any predictive model, and in the same way.
