I have a time series that includes some rare extreme values. We are talking about daily data, in total 1461 observations and 11 extreme values. I adjusted those 11 values with a multiple regression. Now I am using the tbats() on the original time series and the adjusted one.

>                   ME    RMSE      MAE MPE MAPE      MASE          ACF1
>Training set 10.23539 4202.19 2921.593 NaN  Inf 0.6777689 -0.0003493096
>                   ME    RMSE      MAE MPE MAPE      MASE          ACF1
>Training set 43.35625 3803.618 2787.39 NaN  Inf 0.6827622 -0.004749092

#original AIC
#adjusted AIC

How can I see if the model improves due to the adjustment or not? Since I reduced those 11 extreme values, I can't just compare MAE, RMSE or AIC. MASE is the only measure that should work?

I could divide MAE, RMSE and AIC by the mean of the respective time series.

# original
0.4962245 # MAE/mean(original)
0.7137304 # RMSE/mean(original)
5.96188 # AIC/mean(original)

# adjusted
0.4862567 # MAE/mean(adjusted)
0.6635364 # RMSE/mean(adjusted)
6.07051 # AIC/mean(adjusted)

Is that a legitimate way to compare the results?

Here are the pacf-diagrams of both models:






I just realized that when i use the accuracy() function of the forecast package with a tbats() based on a msts() object the resulting MASE is using an in-sample naive forecast for scaling. I guess that is not optimal? It should be better to use an in-sample naive seasonal forecast with the longest season of the msts() object.

MASE(original) # scaled with a in-sample naive seasonal forecast (365)
> 0.6339

MASE(adjusted) # scaled with a in-sample naive seasonal forecast (365)
> 0.6287
  • $\begingroup$ Any thoughts anyone? $\endgroup$ – RandomDude Nov 9 '15 at 9:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.