11
$\begingroup$

Chris Chatfield, whose many quality books and papers I enjoyed reading, in (1) gives the following advice:

For example, the choice between ARIMA time-series models with low and approximately equal values of the AIC should probably be made, not on which happens to give the minimum AIC, but on which gives the best forecasts of the most recent year's data.

What is the rationale for such advice? If it is sound, why does forecast::auto.arima and other forecasting routines do not follow it? Yet to be implemented? It has already been discussed here that to look for models that just happened to give the minimum AIC is probably not a good idea. Why is the option to have $n\ge1$ ARIMA models with low but approximately equal (e.g. within 1 or 2 values of the minimal AIC) is not a default in much of the time series forecasting software?

(1) Chatfield, C. (1991). Avoiding statistical pitfalls. Statistical Science, 6(3), 240–252. Available online, URL: https://projecteuclid.org/euclid.ss/1177011686.

$\endgroup$
  • $\begingroup$ @Gleb_b "An AIC is only high or low in comparison to another." - How can we think differently when we talk about model selection? We always look at low values rather than higher values. What's wrong with my forth sentence? I think it states rather clearly that we are talking about differences (e.g. within one or two of the minimum AIC). There is no mention of absolute values of AIC in the question. $\endgroup$ – Hibernating Jan 9 '14 at 1:56
  • $\begingroup$ I removed "low" from the title and first sentence of the box, for the reasons that @hibernating pointed out. $\endgroup$ – Harvey Motulsky Mar 15 '14 at 13:04
  • $\begingroup$ @Harvey Motulsky Please put "low" back in both places. Thank you. $\endgroup$ – Hibernating Mar 15 '14 at 15:03
  • $\begingroup$ Your question is about how to compare models with the AICs have values fairly close together. High or low is irrelevant (and can change simply by changing the units the data are expressed in). So why put those words back? They are misleading. $\endgroup$ – Harvey Motulsky Mar 15 '14 at 15:54
  • 1
    $\begingroup$ @Harvey Motulsky Please let me be myself. I like the current and my original title "What do I do when values of AIC are low and approximately equal?" I tend to prefer "select" over "choose" in my statistical writing. I have a number of other preferences that characterize me as an individual and they are reflected in the way I form my questions and answers. I am glad you finally understood why I asked you to revert the changes. No problems. $\endgroup$ – Hibernating Mar 17 '14 at 3:11
2
$\begingroup$

It's true that if you have multiple AIC values approximately equal selecting the lowest value may be not the best option. A sensible alternative would be performing model averaging. This way you are able to use not just the best model for inference, but a set of "most supported" models each one weighted according to their AIC value.

You have a short introduction by Vincent Calcagno here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.