0
$\begingroup$

I already know that models from different "families" cannot be compared through AIC (or other information criteria such as BIC or HQIC), however, I'm not sure about the specific case of ARIMA and regression with ARIMA erros. Can you help me with this doubt?

Also if they aren't comparable, can you tell me what is the correct approach to compare the output of both models?

Thanks in advance.

Improve this question
$\endgroup$
5
  • $\begingroup$ Why do you think that models from different "families" cannot be compared through AIC? There have been some heated debates on the question around here, and my take-away is different than yours. $\endgroup$
    – Richard Hardy
    Sep 20, 2019 at 8:55
  • $\begingroup$ @RichardHardy Honestly I don't think I have enough knowledge on the subject to be able to advocate that statement. That is just my current understanding of the use of the AIC from what I have read so far. Apparently, different "families" of models cannot be compared through AIC because their likelihood is computed in different ways (reference: otexts.com/fpp2/arima-ets.html). However, I am willing to read your view on why this is a misconception and, I may even agree with you. Thank you for your help. $\endgroup$
    – Stats Panda
    Sep 20, 2019 at 10:21
  • 1
    $\begingroup$ I think what the cited textbook describes has to do with failing bullet point 3 (and probably bullet point 2) of my answer, which is a software issue rather than a fundamental problem. I will not elaborate on my view this time (that would take considerable effort), but you may always check out the relevant threads on CV if you wish to get some more information on the subject. $\endgroup$
    – Richard Hardy
    Sep 20, 2019 at 10:47
  • $\begingroup$ OK, thank you Richard. $\endgroup$
    – Stats Panda
    Sep 20, 2019 at 11:00
  • 1
    $\begingroup$ You are very welcome! $\endgroup$
    – Richard Hardy
    Sep 20, 2019 at 11:03

1 Answer 1

Reset to default
2
$\begingroup$

ARIMA and regression with ARIMA errors are comparable in terms of AIC as long as

  • the models are estimated by maximum likelihood,
  • the dependent variable is exactly the same and the likelihood is evaluated on exactly the same data points of the dependent variable in all competing models, and
  • the AIC (mainly, the likelihood) is calculated exactly (without scaling, without dropping constants) to ensure comparability across models. (Or the transformations made in AIC calculation are the same across models so that comparability is not compromised.)

The latter point addresses different software implementations which often make the calculated AIC values incomparable.

Alternatively, you can compare the predictive performance of your models using time series cross validation based on rolling or expanding windows.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.