11
$\begingroup$

It is well established, at least among statisticians of some higher calibre, that models with the values of the AIC statistic within a certain threshold of the minimum value should be considered as appropriate as the model minimizing the AIC statistic. For example, in [1, p.221] we find

Then models with small GCV or AIC would be considered best. Of course one should not just blindly minimize GCV or AIC. Rather, all models with reasonably small GCV or AIC values should be considered as potentially appropriate and evaluated according to their simplicity and scientific relevance.

Similarly, in [2, p.144] we have

It has been suggested (Duong, 1984) that models with AIC values within c of the minimum value should be considered competitive (with c=2 as a typical value). Selection from among the competitive models can then be based on such factors as whiteness of the residuals (Section 5.3) and model simplicity.

References:

  1. Ruppert, D.; Wand, M. P. & Carrol, R. J. Semiparametric Regression, Cambridge University Press, 2003
  2. Brockwell, P. J. & Davis, R. A. Introduction to time-series and forecasting, John Wiley & Sons, 1996

So given the above, which of the two models below should be preferred?

print( lh300 <- arima(lh, order=c(3,0,0)) )
# ... sigma^2 estimated as 0.1787:  log likelihood = -27.09,  aic = 64.18
print( lh100 <- arima(lh, order=c(1,0,0)) )
# ... sigma^2 estimated as 0.1975:  log likelihood = -29.38,  aic = 64.76

More generally, when is it appropriate to select models by blindly minimizing the AIC or related statistic?

$\endgroup$
  • $\begingroup$ You haven't given the AIC for either model. $\endgroup$ – Peter Flom - Reinstate Monica Dec 8 '13 at 13:40
  • $\begingroup$ I've shown how to get it with R. $\endgroup$ – Hibernating Dec 8 '13 at 13:56
  • 1
    $\begingroup$ +1 issues in ARIMA models noted below. But otherwise: "Simplifying a prognostic model: a simulation study based on clinical data." Ambler 2002 is the most quoted reference on this. $\endgroup$ – charles Dec 8 '13 at 15:49
3
$\begingroup$

Paraphrasing from Cosma Shalizi’s lecture notes on the truth about Linear Regression, thou shall never choose a model just because it happened to minimise a statistic like AIC, for

Every time someone solely uses an AIC statistic for model selection, an angel loses its
wings. Every time someone thoughtlessly minimises it, an angel not only loses its wings,
but is cast out of Heaven and falls in most extreme agony into the everlasting fire.
$\endgroup$
  • 1
    $\begingroup$ As one famous Jew said: "Imagination is better than knowledge" :) $\endgroup$ – Hibernating Dec 17 '13 at 16:13
  • $\begingroup$ And, as one famous non-Jew said "You can see a lot by looking" (Yogi Berra). $\endgroup$ – Peter Flom - Reinstate Monica Dec 17 '13 at 19:40
  • $\begingroup$ And what we see, of course, depends mainly on what we look for. --John Lubbock $\endgroup$ – Hibernating Dec 18 '13 at 15:51
12
$\begingroup$

I would say it is often appropriate to use AIC in model selection, but rarely right to use it as the sole basis for model selection. We must also use substantive knowledge.

In your particular case, you are comparing a model with a 3rd order AR vs. one with a 1st order AR. In addition to AIC (or something similar) I would look at the the autocorrelation and partial autocorrelation plots. I would also consider what a 3rd order model would mean. Does it make sense? Does it add to substantive knowledge? (Or, if you are solely interested in prediction, does it help predict?)

More generally, it is sometimes the case that finding a very small effect size is interesting.

$\endgroup$
  • $\begingroup$ Did you just say that any good algorithm for selecting an arima model should not be based solely on the AIC (or the like) criterion? $\endgroup$ – Hibernating Dec 8 '13 at 13:54
  • $\begingroup$ Yes I did say that. $\endgroup$ – Peter Flom - Reinstate Monica Dec 8 '13 at 15:00
  • $\begingroup$ And at this end I heard it as goodbye auto.arima. My preference would be to follow an approach outlined in chapter 6 of Bisgaard, S. & Kulahci, M. Time series analysis and forecasting by example John Wiley & Sons, Inc., 2011, even more precisely in section 6.5 IMPULSE RESPONSE FUNCTION TO STUDY THE DIFFERENCES IN MODELS $\endgroup$ – Hibernating Dec 17 '13 at 16:19
  • 1
    $\begingroup$ @Hibernating: The authors of auto.arima, Hyndman & Khandakar (2008), say:- "Automatic forecasts of large numbers of univariate time series are often needed in business. It is common to have over one thousand product lines that need forecasting at least monthly.Even when a smaller number of forecasts are required, there may be nobody suitably trained in the use of time series models to produce them. In these circumstances, an automatic forecasting algorithm is an essential tool." Note these circumstances. $\endgroup$ – Scortchi - Reinstate Monica Jan 11 '14 at 15:03
  • 2
    $\begingroup$ Thanks but I had read that before. Even if we ignore the obvious issues with the "auto" part for now, there are issues with the "arima" part, especially when it is extended to include seasonal models. Seasonal ARIMA models have been strongly criticized by P.J. Harrison, C Chatfield and some other personalities I happened to enjoy learning from. I have nothing against automatic forecasting when it is i) absolutely necessary and ii) based on algorithms I can find sound - otherwise I follow D.R. Cox advice in his comment on Leo Breiman's "two cultures" paper in Stat Science a few years ago. $\endgroup$ – Hibernating Jan 11 '14 at 15:39
8
$\begingroup$

You can think of AIC as a providing a more reasonable (i.e., larger) $P$-value cutoff. But model selection based on $P$-values or any other one-variable-at-a-time metric is frought with difficulties, having all the problems of stepwise variable selection. Generally speaking, AIC works best if used to select a unique single parameter (e.g., shrinkage coefficient) or to compare 2 or 3 candidate models. Otherwise, fitting the entire set of variables in some way, using shrinkage or data reduction, will often result in superior predictive discrimination. Parsimony is at odds with predictive discrimination.

$\endgroup$
  • 2
    $\begingroup$ Your last sentence is interesting. I remember I read that adding even insignificant predictors into regression may well be justified if the ultimate purpose is prediction. I did not pay much attention to it at the time but now I'll try and find that reference. $\endgroup$ – Hibernating Dec 8 '13 at 14:21
  • 2
    $\begingroup$ Instead of adding I would say avoid removing. And it's not just prediction, but using statistical association assessments to guide variable selection causes biases and invalid standard errors and confidence limits. $\endgroup$ – Frank Harrell Dec 8 '13 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.