Should I adjust a Time Series dataset using X-13ARIMA-SEATS before running forecasts algorithms?

Question

This question is sort of a follow up of this great thread.

I have a Time Series Analysis project in which I have to create a model to predict new values given historical univariate data using R. My plan is to run several models, assess their accuracies and choose the best one to use in production.

I must admit that I'm a beginner in TS analysis, so I'm not sure about the procedure that I must take. The main question is: Should I do a seasonal adjust in the data before fitting it into models?

I've done some simulation myself and the results are quite interesting:

Raw data (without adjusting)

library(seasonal)
library(forecast)
data("AirPassengers")

training <- window(AirPassengers, end = c(1959, 12))
test <- window(AirPassengers, start = c(1960, 1))

models <- list(
  mod_arima = auto.arima(training, ic='aicc', stepwise=FALSE),
  mod_etrainingp = ets(training, ic='aicc', restrict=FALSE),
  mod_neural = nnetar(training, p=12, size=25),
  mod_tbats = tbats(training, ic='aicc', seasonal.periods=12),
  mod_bats = bats(training, ic='aicc', seasonal.periods=12),
  mod_stl = stlm(training, s.window=12, ic='aicc', robust=TRUE, method='ets'),
  mod_sts = StructTS(training)
)

forecasts <- lapply(models, forecast, 12)

acc <- lapply(forecasts, function(f){
  accuracy(f, test)[2,,drop=FALSE]
})
acc <- Reduce(rbind, acc)
row.names(acc) <- names(forecasts)
(acc <- round(acc, 2))
                   ME   RMSE    MAE   MPE  MAPE MASE ACF1 Theil's U
mod_arima      -15.72  22.96  17.91 -3.68  4.05 0.59 0.08      0.51
mod_etrainingp   4.99  19.01  14.40  0.75  3.03 0.47 0.27      0.41
mod_neural      12.97  26.08  23.38  2.55  4.94 0.77 0.16      0.55
mod_tbats      -15.49  25.67  18.20 -3.71  4.14 0.60 0.17      0.58
mod_bats         0.69  23.12  16.48 -0.35  3.45 0.54 0.38      0.50
mod_stl         31.35  57.93  39.95  5.31  7.43 1.31 0.65      1.01
mod_sts        177.41 199.41 177.41 36.02 36.02 5.83 0.77      3.76

Adjusted data

adj <- seas(AirPassengers)
adj <- final(adj)

adj_training <- window(adj, end = c(1959, 12))
adj_test <- window(adj, start = c(1960, 1))

models_adj <- list(
  mod_arima = auto.arima(adj_training, ic='aicc', stepwise=FALSE),
  mod_etrainingp = ets(adj_training, ic='aicc', restrict=FALSE),
  mod_neural = nnetar(adj_training, p=12, size=25),
  mod_tbats = tbats(adj_training, ic='aicc', seasonal.periods=12),
  mod_bats = bats(adj_training, ic='aicc', seasonal.periods=12),
  mod_stl = stlm(adj_training, s.window=12, ic='aicc', robust=TRUE, method='ets'),
  mod_sts = StructTS(adj_training)
)

forecasts_adj <- lapply(models_adj, forecast, 12)

acc_adj <- lapply(forecasts_adj, function(f) accuracy(f, adj_test)[2,,drop=FALSE])
acc_adj <- Reduce(rbind, acc_adj)
row.names(acc_adj) <- names(forecasts_adj)
(acc_adj <- round(acc_adj, 2))
                   ME  RMSE   MAE   MPE MAPE MASE ACF1 Theil's U
mod_arima        6.02 10.05  9.49  1.22 1.99 0.31 0.64      1.26
mod_etrainingp  -2.76  7.12  4.43 -0.62 0.96 0.15 0.40      0.90
mod_neural     -30.84 33.27 30.84 -6.47 6.47 1.02 0.56      4.11
mod_tbats      -15.38 16.60 15.38 -3.26 3.26 0.51 0.05      2.09
mod_bats        -7.13 10.41  7.31 -1.55 1.58 0.24 0.43      1.34
mod_stl         -1.77  7.01  4.67 -0.41 1.01 0.15 0.39      0.89
mod_sts         -9.59 11.05  9.59 -2.04 2.04 0.32 0.11      1.38

Now, a comparison table to highlight the differences:

abs(acc_adj) < abs(acc)
                  ME  RMSE   MAE   MPE  MAPE  MASE  ACF1 Theil's U
mod_arima       TRUE  TRUE  TRUE  TRUE  TRUE  TRUE FALSE     FALSE
mod_etrainingp  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE FALSE     FALSE
mod_neural     FALSE FALSE FALSE FALSE FALSE FALSE FALSE     FALSE
mod_tbats       TRUE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE     FALSE
mod_bats       FALSE  TRUE  TRUE FALSE  TRUE  TRUE FALSE     FALSE
mod_stl         TRUE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE      TRUE
mod_sts         TRUE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE      TRUE
as.data.frame(rowMeans(abs(acc_adj) < abs(acc)))
               rowMeans(abs(acc_adj) < abs(acc))
mod_arima                                  0.750
mod_etrainingp                             0.750
mod_neural                                 0.000
mod_tbats                                  0.875
mod_bats                                   0.500
mod_stl                                    1.000
mod_sts                                    1.000

The only model that has better accuracy in raw data is mod_neural, created with neural networks.

So, what should I learn from this experiment? Does it make sense to use X13ARIMA-SEATS to adjust a series before fitting models?

Answer 1

All functions in the forecast package search for the best model according to some evaluation metric (this is usually either AIC, BIC, or AICc). For example, auto.arima essentially performs grid search on all combinations of $P$, $D$ and $Q$ lower than a given threshold, uses these values to perform differencing, then checks the evaluation metric. [BATS and TBATS] take similar approaches, but model seasonality differently (BATS uses an additive approach, while TBATS uses an additive approach in conjunction with trigonometric functions). Some other algorithms use multiplicative decomposition, but I don't think any of them can be found in forecast.

According to the reference manual, X-13 seems to use a combination of seasonal ARIMA and the TRAMO model fitting algorithm to find the right decomposition.

The answer to your initial question, then, depends on the model search algorithm. I am not familiar with TRAMO, but it likely yields a different decomposition than the grid search approach taken by the forecast package's algorithms.

Some other links you might find useful:

• The X-13ARIMA-SEATS vignette.
• An online textbook co-authored by Rob Hyndman, one of the maintainers of forecast, covers the details of ARIMA nicely.

Answer 2

The fundamental issue with what you are suggesting is that you aren't forecasting the same thing. Do you want to forecast the actual data or some seasonally adjusted version of it?

If you really just want to forecast, say, the trend, procedures like X-13 may distort the dynamics of the series, and it might be best to do the forecasting in one step with a single model that contains both trend and seasonality dynamics. On the other hand, the seasonal adjustment part of X-13 (the X-11 procedure) can be viewed as a complicated non-parametric model for seasonality which could potentially better capture past seasonality dynamics. It does not offer a way to forecast this seasonality, but you wouldn't need it if you just wanted to forecast the trend.

Also, the example you have chosen (AirPassengers) is a classic example from the Box-Jenkins methodology, and it is included in the X-13 documentation, presumably because it's a series for which it works very well. It's not exactly the most fair test of your suggestion.