4
$\begingroup$

I'm trying to fit a model to a time series, but I am pretty confused as to which is the best.

I'm looking at an arima model, and ets model and an stlf model, which each performed best within their own family of models. When comparing rolling forecasting errors for 6 month forecasts, they perform exactly equally well, each model has the smallest errors exactly one third of the time.

I then try to look at other criteria such as AIC, AICc and BIC, and get the following results (my problem is really the scale of the information criteria - it's about a factor hundred smaller for the stlf model, is it really that much better or is something else at play here?):

#The arima model:
Series: myts
ARIMA(0,1,0)(1,0,0)[12]
Coefficients:
        sar1
      0.8394
s.e.  0.0704
sigma^2 estimated as 19456:  log likelihood=-229.81
AIC=463.61   AICc=463.99   BIC=466.72

#The ets model:
ETS(M,N,M)
Call:
 ets(y = myts)
  Smoothing parameters:
    alpha = 0.5505
    gamma = 1e-04
  Initial states:
    l = 500.5273
    s=0.5977 0.3134 0.298 0.5218 1.6367 2.0899
           2.1506 2.2123 0.8724 0.5279 0.4086 0.3708

  sigma:  0.1507

     AIC     AICc      BIC
438.9330 458.9330 461.1023

#The stlf model:
 ETS(A,N,N)
Call:
 ets(y = x, model = etsmodel)
  Smoothing parameters:
    alpha = 0.483
  Initial states:
    l = 6.0707
  sigma:  0.1587
      AIC      AICc       BIC
0.4533825 0.8170189 3.6204204

Can they be compared at all? I do think I remember something about only being able to compare these criteria between different models under certain conditions.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Sometimes average AIC (AICc, BIC) is reported: the usual AIC is divided by the number of observation. Check out if multiplying the AIC (AICc, BIC) from stlf by the number of observation will not bring it somewhere close to 400-500 as for the other two methods. Anyhow, I would side with @StephanKolassa in that you should be careful comparing AIC from different algorithms. But sometimes you can track the way AIC is calculated for different algorithms, then there is a chance you could adjust it (if necessary) and compare. $\endgroup$
    – Richard Hardy
    Feb 23, 2015 at 9:49

1 Answer 1

Reset to default
10
$\begingroup$

You can't compare information criteria between different fitting methods. AIC and friends involve a constant that different fitting algorithms set to different values. You can compare AICs for different models fitted by the same method. So no help there.

Looking at rolling out-of-sample forecasts was already exactly the right thing to do. Now you know that each model is best one third of the time. You could now also look at the magnitude of the errors (MAD or MSE) - perhaps one model sometimes yields very low, sometimes very high forecasts.

Failing that, it may well be that all three methods are equally good.

One smart trick to improve forecast accuracy is: calculate forecasts from all three methods and average them within each future time bucket. Averaging forecasts, in particular from "very different" methods, almost always improves accuracy and also reduces error variance.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thanks for clarifying that! How do you go about calculating the variance and confidence intervals of the average of several methods? $\endgroup$
    – SiKiHe
    Feb 23, 2015 at 12:36
  • 1
    $\begingroup$ If your separate methods use normals for their predictive distributions, then "pooling" them amounts to creating a mixture of normals (google that - mixtures are implemented in R and elsewhere), with the point forecast = the expectation of the mixture = the average of the expectations of the components. If your component methods use different predictive distributions (e.g., a normal and a negbin), then your mixture may not be tractable analytically, and you may need to resort to simulation to obtain prediction intervals. $\endgroup$
    – Stephan Kolassa
    Feb 23, 2015 at 12:42
  • 1
    $\begingroup$ (Incidentally, note the difference between a prediction interval, which refers to an interval that contains a future observable realization or not, and a confidence interval, which contains an unobservable parameter or not.) $\endgroup$
    – Stephan Kolassa
    Feb 23, 2015 at 12:43
  • $\begingroup$ I think your answer is imprecise. It says you cannot compare AICs across different fitting methods, the reason being that different methods / software functions calculate AIC values differently (some calculate a modified AIC value). A precise statement should be: for comparability across methods, it is sufficient to use the same definition of AIC. Did I get you right? $\endgroup$
    – Richard Hardy
    Nov 30, 2018 at 20:42
  • $\begingroup$ @RichardHardy: you are right that my answer is imprecise. One aspect I did not mention here is that different R functions or different software package optimize different objective functions in fitting. For instance, Arima() can optimize the likelihood (method="ML"), minimize the constrained sum of squares ("CSS") or a combination ("CSS-ML"). If different criteria are used, it makes little sense to compare likelihoods and likelihood-based information criteria. $\endgroup$
    – Stephan Kolassa
    Dec 1, 2018 at 7:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.