14
$\begingroup$

It is well established, at least among statisticians of some higher calibre, that models with the values of the AIC statistic within a certain threshold of the minimum value should be considered as appropriate as the model minimizing the AIC statistic. For example, in [1, p.221] we find

Then models with small GCV or AIC would be considered best. Of course one should not just blindly minimize GCV or AIC. Rather, all models with reasonably small GCV or AIC values should be considered as potentially appropriate and evaluated according to their simplicity and scientific relevance.

Similarly, in [2, p.144] we have

It has been suggested (Duong, 1984) that models with AIC values within c of the minimum value should be considered competitive (with c=2 as a typical value). Selection from among the competitive models can then be based on such factors as whiteness of the residuals (Section 5.3) and model simplicity.

References:

  1. Ruppert, D.; Wand, M. P. & Carrol, R. J. Semiparametric Regression, Cambridge University Press, 2003
  2. Brockwell, P. J. & Davis, R. A. Introduction to time-series and forecasting, John Wiley & Sons, 1996

So given the above, which of the two models below should be preferred?

print( lh300 <- arima(lh, order=c(3,0,0)) )
# ... sigma^2 estimated as 0.1787:  log likelihood = -27.09,  aic = 64.18
print( lh100 <- arima(lh, order=c(1,0,0)) )
# ... sigma^2 estimated as 0.1975:  log likelihood = -29.38,  aic = 64.76

More generally, when is it appropriate to select models by blindly minimizing the AIC or related statistic?

$\endgroup$
3
  • 1
    $\begingroup$ You haven't given the AIC for either model. $\endgroup$
    – Peter Flom
    Dec 8, 2013 at 13:40
  • $\begingroup$ I've shown how to get it with R. $\endgroup$ Dec 8, 2013 at 13:56
  • 1
    $\begingroup$ +1 issues in ARIMA models noted below. But otherwise: "Simplifying a prognostic model: a simulation study based on clinical data." Ambler 2002 is the most quoted reference on this. $\endgroup$
    – charles
    Dec 8, 2013 at 15:49

3 Answers 3

8
$\begingroup$

Paraphrasing from Cosma Shalizi’s lecture notes on the truth about Linear Regression, thou shall never choose a model just because it happened to minimise a statistic like AIC, for

Every time someone solely uses an AIC statistic for model selection, an angel loses its wings. Every time someone thoughtlessly minimises it, an angel not only loses its wings, but is cast out of Heaven and falls in most extreme agony into the everlasting fire.

$\endgroup$
0
16
$\begingroup$

I would say it is often appropriate to use AIC in model selection, but rarely right to use it as the sole basis for model selection. We must also use substantive knowledge.

In your particular case, you are comparing a model with a 3rd order AR vs. one with a 1st order AR. In addition to AIC (or something similar) I would look at the the autocorrelation and partial autocorrelation plots. I would also consider what a 3rd order model would mean. Does it make sense? Does it add to substantive knowledge? (Or, if you are solely interested in prediction, does it help predict?)

More generally, it is sometimes the case that finding a very small effect size is interesting.

$\endgroup$
5
  • $\begingroup$ Did you just say that any good algorithm for selecting an arima model should not be based solely on the AIC (or the like) criterion? $\endgroup$ Dec 8, 2013 at 13:54
  • $\begingroup$ Yes I did say that. $\endgroup$
    – Peter Flom
    Dec 8, 2013 at 15:00
  • $\begingroup$ And at this end I heard it as goodbye auto.arima. My preference would be to follow an approach outlined in chapter 6 of Bisgaard, S. & Kulahci, M. Time series analysis and forecasting by example John Wiley & Sons, Inc., 2011, even more precisely in section 6.5 IMPULSE RESPONSE FUNCTION TO STUDY THE DIFFERENCES IN MODELS $\endgroup$ Dec 17, 2013 at 16:19
  • 2
    $\begingroup$ @Hibernating: The authors of auto.arima, Hyndman & Khandakar (2008), say:- "Automatic forecasts of large numbers of univariate time series are often needed in business. It is common to have over one thousand product lines that need forecasting at least monthly.Even when a smaller number of forecasts are required, there may be nobody suitably trained in the use of time series models to produce them. In these circumstances, an automatic forecasting algorithm is an essential tool." Note these circumstances. $\endgroup$ Jan 11, 2014 at 15:03
  • 2
    $\begingroup$ Thanks but I had read that before. Even if we ignore the obvious issues with the "auto" part for now, there are issues with the "arima" part, especially when it is extended to include seasonal models. Seasonal ARIMA models have been strongly criticized by P.J. Harrison, C Chatfield and some other personalities I happened to enjoy learning from. I have nothing against automatic forecasting when it is i) absolutely necessary and ii) based on algorithms I can find sound - otherwise I follow D.R. Cox advice in his comment on Leo Breiman's "two cultures" paper in Stat Science a few years ago. $\endgroup$ Jan 11, 2014 at 15:39
9
$\begingroup$

You can think of AIC as a providing a more reasonable (i.e., larger) $P$-value cutoff. But model selection based on $P$-values or any other one-variable-at-a-time metric is frought with difficulties, having all the problems of stepwise variable selection. Generally speaking, AIC works best if used to select a unique single parameter (e.g., shrinkage coefficient) or to compare 2 or 3 candidate models. Otherwise, fitting the entire set of variables in some way, using shrinkage or data reduction, will often result in superior predictive discrimination. Parsimony is at odds with predictive discrimination.

$\endgroup$
2
  • 2
    $\begingroup$ Your last sentence is interesting. I remember I read that adding even insignificant predictors into regression may well be justified if the ultimate purpose is prediction. I did not pay much attention to it at the time but now I'll try and find that reference. $\endgroup$ Dec 8, 2013 at 14:21
  • 3
    $\begingroup$ Instead of adding I would say avoid removing. And it's not just prediction, but using statistical association assessments to guide variable selection causes biases and invalid standard errors and confidence limits. $\endgroup$ Dec 8, 2013 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.