I have applied Exponential Smoothing methods on data (Quarterly electricity production in Australia million kilowatt hourly) and then I forecast the accuracy of my models,

ep_m<-msts(scan("e:\\DATA\\Quarterly electricity production in Australia million kilowatt hour.csv"),seasonal.periods = 4 )


ets(y = ep_m, model = "MMM") 

  Smoothing parameters:
    alpha = 0.5751 
    beta  = 0.0424 
    gamma = 0.3636 

  Initial states:
    l = 4168.6162 
    b = 1.0194 
    s=0.9641 1.0806 1.0286 0.9267

  sigma:  0.0164

     AIC         AICc           BIC 
2531.925    2532.911    2556.272 



ets(y = ep_m, model = "MAM")

  Smoothing parameters:
    alpha = 0.5574 
    beta  = 0.0634 
    gamma = 0.3732 

  Initial states:
    l = 4172.3008 
    b = 90.599 
    s=0.9644 1.0815 1.025 0.9291

  sigma:  0.0165

     AIC                AICc                  BIC 
2533.216     2534.202       2557.563

                      ME           RMSE            MAE               MPE                 MAPE           MASE            ACF1
Training set -69.33972   396.3885     274.0834     -0.2291014     1.316194     0.1901402    0.008993804

                     ME             RMSE            MAE               MPE                 MAPE           MASE             ACF1
Training set   1.494434   385.2168     269.2909      0.1453125      1.306918      0.1868155   -0.008601048

My Question: If I compare the two models with AIC-BIC the best model is "MMM" but if I compare with MAPE-MASE the best model is "MAM".

Which model I have to choose and why to choose it ?

I tried to find the best forecasting accuracy but I didn't find anything helpful.

  • 1
    $\begingroup$ Please take a look at: stackoverflow.com/help/how-to-ask $\endgroup$ – mcNets Nov 21 '16 at 19:34
  • $\begingroup$ Have you read the answer? I see you have neither accepted nor upvoted it. Is anything unclear? $\endgroup$ – Richard Hardy Nov 27 '16 at 16:24
  • $\begingroup$ Thank you Mr.@RichardHardy for your helpful answer. I couldn't reach the web since a week. I convinced that in the same class we have to compare using AIC and BIC. But when comparing forecasting models not in the same class we must use MAPE or MASE. $\endgroup$ – OmarMH87 Nov 30 '16 at 17:46
  • $\begingroup$ Thanks. Was that a question? If so, why do you think so? $\endgroup$ – Richard Hardy Nov 30 '16 at 21:11
  • $\begingroup$ Mr. @RichardHardy, I appreciate your help. And the last comment isn't a question. It's an notice I figured it from "Automatic Time Series Forecasting: The forecast Package for R". AIC provides a method for selecting between additive and multiplicative error methods because it is based on likelihood rather than one-step forecasts. $\endgroup$ – OmarMH87 Dec 2 '16 at 21:46

Use AIC or BIC rather than MAPE or MASE from the training set. Here is why:

The measures on the training set (training sample) are not really suitable as basis for model selection. It is because in the training sample it is always possible to overfit, and the richer the model, the better the fit will be.

Meanwhile, information criteria like AIC or BIC take this into account and penalize for the model complexity accordingly. Therefore, they generally are suitable for model selection.

One other possibility is to assess measures of fit such as MAPE or MASE on a test sample. This way you avoid overfitting because the models are fit on the training sample but evaluated on the test sample.

There is some relation between the latter two approaches. For example, asymptotically AIC should select the model that has the lowest 1-step-ahead RMSE on a test sample.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.