I already know that models from different "families" cannot be compared through AIC (or other information criteria such as BIC or HQIC), however, I'm not sure about the specific case of ARIMA and regression with ARIMA erros. Can you help me with this doubt?

Also if they aren't comparable, can you tell me what is the correct approach to compare the output of both models?

Thanks in advance.

  • $\begingroup$ Why do you think that models from different "families" cannot be compared through AIC? There have been some heated debates on the question around here, and my take-away is different than yours. $\endgroup$ – Richard Hardy Sep 20 '19 at 8:55
  • $\begingroup$ @RichardHardy Honestly I don't think I have enough knowledge on the subject to be able to advocate that statement. That is just my current understanding of the use of the AIC from what I have read so far. Apparently, different "families" of models cannot be compared through AIC because their likelihood is computed in different ways (reference: otexts.com/fpp2/arima-ets.html). However, I am willing to read your view on why this is a misconception and, I may even agree with you. Thank you for your help. $\endgroup$ – Stats Panda Sep 20 '19 at 10:21
  • 1
    $\begingroup$ I think what the cited textbook describes has to do with failing bullet point 3 (and probably bullet point 2) of my answer, which is a software issue rather than a fundamental problem. I will not elaborate on my view this time (that would take considerable effort), but you may always check out the relevant threads on CV if you wish to get some more information on the subject. $\endgroup$ – Richard Hardy Sep 20 '19 at 10:47
  • $\begingroup$ OK, thank you Richard. $\endgroup$ – Stats Panda Sep 20 '19 at 11:00
  • 1
    $\begingroup$ You are very welcome! $\endgroup$ – Richard Hardy Sep 20 '19 at 11:03

ARIMA and regression with ARIMA errors are comparable in terms of AIC as long as

  • the models are estimated by maximum likelihood,
  • the dependent variable is exactly the same and the likelihood is evaluated on exactly the same data points of the dependent variable in all competing models, and
  • the AIC (mainly, the likelihood) is calculated exactly (without scaling, without dropping constants) to ensure comparability across models. (Or the transformations made in AIC calculation are the same across models so that comparability is not compromised.)

The latter point addresses different software implementations which often make the calculated AIC values incomparable.

Alternatively, you can compare the predictive performance of your models using time series cross validation based on rolling or expanding windows.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.