I have a time series that includes some rare extreme values. We are talking about daily data, in total 1461 observations and 11 extreme values. I adjusted those 11 values with a multiple regression. Now I am using the
tbats() on the original time series and the adjusted one.
accuracy(original) > ME RMSE MAE MPE MAPE MASE ACF1 >Training set 10.23539 4202.19 2921.593 NaN Inf 0.6777689 -0.0003493096 accuracy(adjusted) > ME RMSE MAE MPE MAPE MASE ACF1 >Training set 43.35625 3803.618 2787.39 NaN Inf 0.6827622 -0.004749092 #original AIC >35101.43 #adjusted AIC >34798.24
How can I see if the model improves due to the adjustment or not? Since I reduced those 11 extreme values, I can't just compare MAE, RMSE or AIC. MASE is the only measure that should work?
I could divide MAE, RMSE and AIC by the mean of the respective time series.
# original 0.4962245 # MAE/mean(original) 0.7137304 # RMSE/mean(original) 5.96188 # AIC/mean(original) # adjusted 0.4862567 # MAE/mean(adjusted) 0.6635364 # RMSE/mean(adjusted) 6.07051 # AIC/mean(adjusted)
Is that a legitimate way to compare the results?
Here are the
pacf-diagrams of both models:
I just realized that when i use the
accuracy() function of the
forecast package with a
tbats() based on a
msts() object the resulting MASE is using an in-sample naive forecast for scaling. I guess that is not optimal? It should be better to use an in-sample naive seasonal forecast with the longest season of the
MASE(original) # scaled with a in-sample naive seasonal forecast (365) > 0.6339 MASE(adjusted) # scaled with a in-sample naive seasonal forecast (365) > 0.6287