As far as I understand information criteria (IC) can be used to perform hyper parameter tuning/variable selection (e.g. I can use AIC to find the best $p^*$ and $q^*$ for an ARMA$(p,q)$ model) but not to select models from 'different classes': e.g. I cannot compare the AIC values of an ARIMA$(1,1,1)$ with that of an ARIMA$(1,2,1)$ because of the different order of differencing nor can I compare an ARMA with a ETS (as per point 5 here for example http://robjhyndman.com/hyndsight/aic/).

1) How exactly can one discriminate between what can be compared through IC and what cannot? (For instance, can I safely compare ARMA with GARCH?)

2) Is it not incorrect then to refer to IC as methods to perform model selection? Should we not rather refer to them as methods for variable selection?

3) If this is the case, what other methods for model selection are we left with apart from all the variations of CV?

  • $\begingroup$ It is not clear to me what the varying abbreviations you used stand for. Could you write them down in full? $\endgroup$ – IWS Jan 19 '17 at 13:30
  • $\begingroup$ These I believe are pretty standard abbreviations, but if you are interested: CV is for cross validation, AIC for Akaike information creterion, ARMA autoregressive moving average models, as GARCH and ETS are other well known models. $\endgroup$ – semola Jan 19 '17 at 18:59

Information criteria could be used for model selection as for example BIC in the limit of large amount of data approximates marginal likelihood (up to a constant). And AIC (under some assumptions) tells you about the predictive power of the model. Obviously all of those are approximations and will be inappropriate in some cases, while cross-validation is pretty generic. There are however other information criteria that approximate the cross-validation better (i.e. WAIC) See this paper for more discussion http://www.stat.columbia.edu/~gelman/research/published/waic_understand3.pdf

| cite | improve this answer | |
  • $\begingroup$ Thank you, but unfortunately this does not answer my question in any way. I am aware of what AIC and BIC are. $\endgroup$ – semola Jan 23 '17 at 0:11
  • $\begingroup$ Then you may want to rephrase/expand the question and explain what you meant by "be considered model selection methods in the strict sense" $\endgroup$ – sega_sai Jan 23 '17 at 0:16
  • $\begingroup$ It is explained above in the question: you can -for example- use AIC to select between and ARMA(2,3) and ARMA(2,4) but not to select between ARMA(2,3) and ETS. Nor you can use it to compare select between ARIMA(1,2,3) and ARIMA(1,3,3). Reference here: robjhyndman.com/hyndsight/aic $\endgroup$ – semola Jan 23 '17 at 0:23
  • $\begingroup$ Okay, I wasn't very aware of ARIMA/ETS specifics. Now I think I see what's going on. Basically what is referred to as AIC in the context of ARIMA/ETS is not quite the AIC because it using the conditional likelihoods, rather than the full/exact likelihoods. If those would have been used instead, then you could have compared AIC of all those different models. So the issue is not that AIC cannot be used for model selection, it could, it is just if the AIC have to be computed on the whole data rather being conditioned on part of it. $\endgroup$ – sega_sai Jan 23 '17 at 2:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.