7
$\begingroup$

On scikit-learn documentation, I found the following comments about AIC:

Information-criterion based model selection is very fast, but it relies on a proper estimation of degrees of freedom, are derived for large samples (asymptotic results) and assume the model is correct, i.e. that the data are actually generated by this model. They also tend to break when the problem is badly conditioned (more features than samples).

My questions are:

  1. Why would AIC break when we have more features than samples?
  2. Why is AIC and BIC commonly used in forecasting model like ARIMA?
$\endgroup$
2
7
$\begingroup$

What alternatives do we have in model selection for prediction?

  • The main ones are cross validation and information criteria.

Why are the latter attractive in the time series setting?

  • Information criteria are less computationally intensive. You only need to fit the model once to calculate an information criterion. This is in contrast to most applications of cross validation. Computational efficience is extra desirable in the time series setting as many basic time series models (ARMA, GARCH and the like) tend to be rather computationally demanding (more so than, say, linear regression).
  • Information criteria are also more effective in utilizing the data, as the model is estimated on the entire sample rather than just a training subset. The latter is important in small data sets* and especially in time series settings. In small data sets, we do not want to leave out too much data for testing, as then there is very little data left for training/estimation. We have leave-one-out cross validation (LOOCV) which leaves out only a single observation at a time in training/estimation, and it works well in a cross-sectional setting. However, it is often inapplicable in the time series setting due to the mutual dependence of the observations. Other types of validation that are applicable are much more data-costly. For more details, see "AIC versus cross validation in time series: the small sample case".

*Information criteria have an asymptotic justification, so their use is not unproblematic in small samples. Nevetherless, a more efficient use of the data is more desirable than a less efficient use. By using the entire sample for estimation you are closer to asymptotics than by using, say, 2/3 of the sample.

$\endgroup$
6
$\begingroup$

First off, as Richard Hardy comments, information criteria do not assume we have the true model. Quite to the contrary. For instance, AIC estimates the Kullback-Leibler distance between the proposed model and the true data generating process (up to an offset), and picking the model with minimal AIC amounts to choosing the one with the smallest distance to the true DGP. See Burnham & Anderson (2002, Model selection and multi-model inference: a practical information-theoretic approach) or Burnham & Anderson (2004, Sociological Methods & Research) for an accessible treatment. They also go into the justification for BIC.

Information criteria break down with overparameterized models, but that's not really a problem of the ICs. Instead, it's that every overparameterized model that is not regularized breaks down, and that "normal" ICs don't work with regularized models. (I believe there are IC variants that apply to regularized models, but am not an expert in this.)

ICs are used in forecasting model selection because of the above argument about distances to true DGPs. A related argument is that the AIC asymptotically estimates a monotone function of the prediction error (section 4.3.1 in Lütkepohl, 2005, New Introduction to Multiple Time Series Analysis, who also goes into other model selection criteria). Also, ICs are not the only tool used: some people prefer using holdout sets, but that means you need more data.

$\endgroup$
2
  • $\begingroup$ Regarding AIC asymptotically estimates a monotone function of the prediction error, I have a related question: "Equivalence of AIC and LOOCV under mismatched loss functions" (and more related questions linked in there). $\endgroup$ – Richard Hardy May 10 at 8:24
  • $\begingroup$ I think the primary reason is that there are few if any good alternatives. The real issue is which information criterion you use, AIC, BIC, etc. There are differing opinions on that. Note that some use hold out data sets (and MAPE, MSE etc)to choose which model is best, I do, but that is not considered a statistical approach I assume. $\endgroup$ – user54285 May 10 at 23:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.