Why TBATS model giving poor result?

Question

I have time series data of number of units ordered from a manufacturing plant and number of units delivered. The are multiple different plant sites for which I need to build forecasting models. I started with one plant and used ARIMA models with seasonality and differencing of order 1 to induce stationarity. The model performance was decent but not very good. Here is the R code

model = auto.arima(log(train),d=1, seasonal = T)

f = forecast(model, 10)
p = exp(f$mean) 
mape(test,p)*100

At this stage I had monthly data and number of data points was only 31 for each plant. Due to fewer data points I could not try other approaches. To improve the model I collected weekly data and tried other approaches. Since I had weekly data, I thought of modelling complex seasonal patterns using TBATS. Here is the R code

y <- msts(train_a, seasonal.periods=12)
fit <- tbats(y) 
fc <- forecast(fit,h=5001)

To my surprise, I got insanely high MAPE using this method, which made me feel I am doing something wrong. I went back to try ARIMA again on weekly data. However, I noticed that if I used log(train_a) to train the model I get the error which says "No suitable ARIMA model found". And if I use train_a to train the model I get almost the same high MAPE I got with TBATS.

1. I am not sure why using weekly data (around 2500 data points) is giving very high error but using monthly data points (only 31 data points) was giving decent result (although not very good). What am I doing wrong here ?

2. Given the result with TBATS was very poor, how can I check if TBATS is the appropriate model to use?

3. When should I consider going for additive time series models such as those of prophet library?

4. What could be some of the ways I can look at to improve the prediction accuracy?

Answer 1

1. I am not sure why using weekly data (around 2500 data points) is giving very high error but using monthly data points (only 31 data points) was giving decent result (although not very good). What am I doing wrong here ?

@IrishStat is likely right. The 53rd week issue is rough.

It is usually easier to get clear seasonality with monthly data or daily data. If you want to forecast at the monthly level one option is to take monthly values then divide the monthly forecast into weekly forecasts using proportions from last years data. This is very adhoc, if there is a large trend this method would likely not work.

If possible I prefer to work with daily data and aggregate up. Daily data usually has weekly seasonality and yearly seasonality. You asked about prophet. I have seen good results on daily data with prophet.

2. Given the result with TBATS was very poor, how can I check if TBATS is the appropriate model to use?

In general comparing AIC or having a hold-out test set is the way forecasters determine what to use. I am not sure why TBATs is giving poor results. You can always try using ARIMA with fourier regressors if you want something analogous to TBATS.

3. When should I consider going for additive time series models such as those of prophet library?

Daily data with multiple seasonalities.

4. What could be some of the ways I can look at to improve the prediction accuracy?

1. Always use a benchmark!
2. Combine forecasts from multiple methods

If I assume that your data has seasonality of 4, thinking that week 1 of month 12 is like week 1 of month 11, then these are the results I get.

tbats

                  ME     RMSE      MAE       MPE     MAPE      MASE       ACF1 Theil's U
Training set 303940.9 992404.8 660849.8 -3.715019 33.47303 1.0359822 -0.2295624        NA
Test set     127521.3 663545.9 491123.4 -8.246457 32.40637 0.7699103 -0.4565056 0.2511831

arima

                    ME     RMSE      MAE        MPE     MAPE      MASE       ACF1 Theil's U
Training set  176783.7 914286.5 557117.1 -0.8916543 24.60861 0.8733655 -0.1641036        NA
Test set     -169576.8 395043.5 304737.5 -8.7959001 16.77696 0.4777222  0.1541620 0.1634051

arima with fourier

                     ME     RMSE      MAE       MPE     MAPE     MASE       ACF1 Theil's U
Training set  -26492.66 854052.9 684739.0 -17.12327 38.64117 1.073432  0.1110644        NA
Test set     -290873.37 735399.4 642696.9 -36.66567 49.56333 1.007525 -0.3913094 0.6469734

seasonal naive

                     ME     RMSE      MAE        MPE     MAPE     MASE       ACF1 Theil's U
Training set  -19057.05 949002.6 637896.8  -6.941991 28.86181 1.000000 -0.4136366        NA
Test set     -155011.50 167192.4 155011.5 -10.319070 10.31907 0.243004  0.2955041 0.1352975

You will see that using seasonal naive on a hold out of 4 weeks with the assumption of seasonality of 4 give the best results on the test set!

Code

library(forecast)
df_weekly <- c(600188, 1474332, 1764480, 4501748, 638978, 1410116, 2359624, 3577692, 1540400, 
               2096944, 1729844, 2046916, 4479828, 1378832, 1360384, 1562646, 3546402, 1003902, 
               1273794, 1646558, 3490476, 1041898, 1509698, 1762512, 2085656, 3733219, 1314034, 
               1472608, 1782192, 2350811, 1367302, 1501768, 1396170, 3285756, 1073122, 1433666, 
               1916046, 1913576, 2967130, 1083428, 1791398, 1774624, 2727810, 894408, 1680798, 
               1693518, 2303190)

ts_weekly <- ts(df_weekly, frequency = 4)

train <- subset(ts_weekly, end=length(ts_weekly)-5)
test <- subset(ts_weekly, start=length(ts_weekly)-4)

benchmark <- snaive(train) %>% forecast(h = 4)
tbats_fit <- tbats(train, seasonal.periods = 52) %>% forecast(h = 4)
arima_fit <- auto.arima(train, lambda = "auto") %>% forecast(h = 4)
# arima with fourier regs
bestfit <- list(aicc=Inf)
for(i in 1:8)
{
  fit <- auto.arima(train, xreg=fourier(train, K=i), seasonal=FALSE)
  if(fit$aicc < bestfit$aicc)
    bestfit <- fit
  else break;
}
arimaf_fit <- forecast(bestfit, xreg=fourier(train, K=1, h=4))

accuracy(tbats_fit, test) 
accuracy(arima_fit, test)
accuracy(arimaf_fit, test)
accuracy(benchmark, test)
```

Answer 2

I used the 44 values . Visually there appears to be more variability at higher values . This lead the automatic analysis to suggest a log xform using AUTOBOX , a piece of software that I have helped to develop.

Following is the Actual/Fit and Forecast from a useful model in log space (3,0,0)(0,0,0)

The equation is here and here

The model residuals are here with forecasts here

Since you only have 44 values it is not possible to develop seasonal structure .

As to why TBATS is giving you unacceptable results .. I can not help you . Perhaps it didn't sense the need for a power transform or was thwarted by the anomalies .. I don't know.