I have a time series that includes some rare extreme values. We are talking about daily data, in total 1461 observations and 11 extreme values. I adjusted those 11 values with a multiple regression. Now I am using the tbats() on the original time series and the adjusted one.

>                   ME    RMSE      MAE MPE MAPE      MASE          ACF1
>Training set 10.23539 4202.19 2921.593 NaN  Inf 0.6777689 -0.0003493096
>                   ME    RMSE      MAE MPE MAPE      MASE          ACF1
>Training set 43.35625 3803.618 2787.39 NaN  Inf 0.6827622 -0.004749092

#original AIC
#adjusted AIC

How can I see if the model improves due to the adjustment or not? Since I reduced those 11 extreme values, I can't just compare MAE, RMSE or AIC. MASE is the only measure that should work?

I could divide MAE, RMSE and AIC by the mean of the respective time series.

# original
0.4962245 # MAE/mean(original)
0.7137304 # RMSE/mean(original)
5.96188 # AIC/mean(original)

# adjusted
0.4862567 # MAE/mean(adjusted)
0.6635364 # RMSE/mean(adjusted)
6.07051 # AIC/mean(adjusted)

Is that a legitimate way to compare the results?

Here are the pacf-diagrams of both models:






I just realized that when i use the accuracy() function of the forecast package with a tbats() based on a msts() object the resulting MASE is using an in-sample naive forecast for scaling. I guess that is not optimal? It should be better to use an in-sample naive seasonal forecast with the longest season of the msts() object.

MASE(original) # scaled with a in-sample naive seasonal forecast (365)
> 0.6339

MASE(adjusted) # scaled with a in-sample naive seasonal forecast (365)
> 0.6287
  • $\begingroup$ Any thoughts anyone? $\endgroup$
    – RandomDude
    Commented Nov 9, 2015 at 9:14


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.