1
$\begingroup$

This question is sort of a follow up of this great thread.

I have a Time Series Analysis project in which I have to create a model to predict new values given historical univariate data using R. My plan is to run several models, assess their accuracies and choose the best one to use in production.

I must admit that I'm a beginner in TS analysis, so I'm not sure about the procedure that I must take. The main question is: Should I do a seasonal adjust in the data before fitting it into models?

I've done some simulation myself and the results are quite interesting:

Raw data (without adjusting)

library(seasonal)
library(forecast)
data("AirPassengers")

training <- window(AirPassengers, end = c(1959, 12))
test <- window(AirPassengers, start = c(1960, 1))

models <- list(
  mod_arima = auto.arima(training, ic='aicc', stepwise=FALSE),
  mod_etrainingp = ets(training, ic='aicc', restrict=FALSE),
  mod_neural = nnetar(training, p=12, size=25),
  mod_tbats = tbats(training, ic='aicc', seasonal.periods=12),
  mod_bats = bats(training, ic='aicc', seasonal.periods=12),
  mod_stl = stlm(training, s.window=12, ic='aicc', robust=TRUE, method='ets'),
  mod_sts = StructTS(training)
)

forecasts <- lapply(models, forecast, 12)

acc <- lapply(forecasts, function(f){
  accuracy(f, test)[2,,drop=FALSE]
})
acc <- Reduce(rbind, acc)
row.names(acc) <- names(forecasts)
(acc <- round(acc, 2))
                   ME   RMSE    MAE   MPE  MAPE MASE ACF1 Theil's U
mod_arima      -15.72  22.96  17.91 -3.68  4.05 0.59 0.08      0.51
mod_etrainingp   4.99  19.01  14.40  0.75  3.03 0.47 0.27      0.41
mod_neural      12.97  26.08  23.38  2.55  4.94 0.77 0.16      0.55
mod_tbats      -15.49  25.67  18.20 -3.71  4.14 0.60 0.17      0.58
mod_bats         0.69  23.12  16.48 -0.35  3.45 0.54 0.38      0.50
mod_stl         31.35  57.93  39.95  5.31  7.43 1.31 0.65      1.01
mod_sts        177.41 199.41 177.41 36.02 36.02 5.83 0.77      3.76

Adjusted data

adj <- seas(AirPassengers)
adj <- final(adj)

adj_training <- window(adj, end = c(1959, 12))
adj_test <- window(adj, start = c(1960, 1))

models_adj <- list(
  mod_arima = auto.arima(adj_training, ic='aicc', stepwise=FALSE),
  mod_etrainingp = ets(adj_training, ic='aicc', restrict=FALSE),
  mod_neural = nnetar(adj_training, p=12, size=25),
  mod_tbats = tbats(adj_training, ic='aicc', seasonal.periods=12),
  mod_bats = bats(adj_training, ic='aicc', seasonal.periods=12),
  mod_stl = stlm(adj_training, s.window=12, ic='aicc', robust=TRUE, method='ets'),
  mod_sts = StructTS(adj_training)
)

forecasts_adj <- lapply(models_adj, forecast, 12)

acc_adj <- lapply(forecasts_adj, function(f) accuracy(f, adj_test)[2,,drop=FALSE])
acc_adj <- Reduce(rbind, acc_adj)
row.names(acc_adj) <- names(forecasts_adj)
(acc_adj <- round(acc_adj, 2))
                   ME  RMSE   MAE   MPE MAPE MASE ACF1 Theil's U
mod_arima        6.02 10.05  9.49  1.22 1.99 0.31 0.64      1.26
mod_etrainingp  -2.76  7.12  4.43 -0.62 0.96 0.15 0.40      0.90
mod_neural     -30.84 33.27 30.84 -6.47 6.47 1.02 0.56      4.11
mod_tbats      -15.38 16.60 15.38 -3.26 3.26 0.51 0.05      2.09
mod_bats        -7.13 10.41  7.31 -1.55 1.58 0.24 0.43      1.34
mod_stl         -1.77  7.01  4.67 -0.41 1.01 0.15 0.39      0.89
mod_sts         -9.59 11.05  9.59 -2.04 2.04 0.32 0.11      1.38

Now, a comparison table to highlight the differences:

abs(acc_adj) < abs(acc)
                  ME  RMSE   MAE   MPE  MAPE  MASE  ACF1 Theil's U
mod_arima       TRUE  TRUE  TRUE  TRUE  TRUE  TRUE FALSE     FALSE
mod_etrainingp  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE FALSE     FALSE
mod_neural     FALSE FALSE FALSE FALSE FALSE FALSE FALSE     FALSE
mod_tbats       TRUE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE     FALSE
mod_bats       FALSE  TRUE  TRUE FALSE  TRUE  TRUE FALSE     FALSE
mod_stl         TRUE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE      TRUE
mod_sts         TRUE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE      TRUE
as.data.frame(rowMeans(abs(acc_adj) < abs(acc)))
               rowMeans(abs(acc_adj) < abs(acc))
mod_arima                                  0.750
mod_etrainingp                             0.750
mod_neural                                 0.000
mod_tbats                                  0.875
mod_bats                                   0.500
mod_stl                                    1.000
mod_sts                                    1.000

The only model that has better accuracy in raw data is mod_neural, created with neural networks.

So, what should I learn from this experiment? Does it make sense to use X13ARIMA-SEATS to adjust a series before fitting models?

$\endgroup$
1
  • $\begingroup$ AFAIK, 1.First you have to determine those models are comparable or not. 2. X13ARIMA-SEATS has it's own algorithm to parse the time series. So, you can totally trust it and consider the forecasts automatically generated through this as an estimate. $\endgroup$ Jul 3, 2016 at 6:54

2 Answers 2

2
$\begingroup$

All functions in the forecast package search for the best model according to some evaluation metric (this is usually either AIC, BIC, or AICc). For example, auto.arima essentially performs grid search on all combinations of $P$, $D$ and $Q$ lower than a given threshold, uses these values to perform differencing, then checks the evaluation metric. [BATS and TBATS] take similar approaches, but model seasonality differently (BATS uses an additive approach, while TBATS uses an additive approach in conjunction with trigonometric functions). Some other algorithms use multiplicative decomposition, but I don't think any of them can be found in forecast.

According to the reference manual, X-13 seems to use a combination of seasonal ARIMA and the TRAMO model fitting algorithm to find the right decomposition.

The answer to your initial question, then, depends on the model search algorithm. I am not familiar with TRAMO, but it likely yields a different decomposition than the grid search approach taken by the forecast package's algorithms.

Some other links you might find useful:

$\endgroup$
3
  • 2
    $\begingroup$ That's not how X13 works. It uses a regression with (seasonal) ARIMA errors model to remove certain effects (outliers, moving holidays, trading days, etc.) and to forecast a year out. Then, it uses a series of moving averages (called the "X11" procedure originally) to estimate and remove seasonality (the 1-year forecast is used by the MA's at the end of the series). TRAMO-SEATS is an alternate algorithm that is included in the software for comparison, but AFAIK not used by default. $\endgroup$
    – Chris Haug
    Nov 6, 2016 at 17:31
  • $\begingroup$ I found this in the reference manual, re: X-13 automated model selection: "In addition to these modeling features, X-13ARIMA-SEATS has an automatic model selection procedure based largely on the automatic model selection procedure of TRAMO (Gomez and Maravall 1996, documented in Gomez and Maravall 2001a)." $\endgroup$
    – achompas
    Nov 6, 2016 at 21:44
  • 1
    $\begingroup$ Ok, correct. The automdl spec in X-13 is an information criterion-based model selection procedure based on TRAMO. It is not a default in X-13 itself but I think it may be if you're using the R seasonal interface to X-13. My point was that this model is not used for decomposition, it is used for forecasting at the end so that symmetric moving average filters may be used for decomposition. It's really just SEATS which is the alternate Bank of Spain procedure for adjustment that was added in X-13, I forgot that the TRAMO part was already in there in some form. $\endgroup$
    – Chris Haug
    Nov 6, 2016 at 22:55
2
$\begingroup$

The fundamental issue with what you are suggesting is that you aren't forecasting the same thing. Do you want to forecast the actual data or some seasonally adjusted version of it?

If you really just want to forecast, say, the trend, procedures like X-13 may distort the dynamics of the series, and it might be best to do the forecasting in one step with a single model that contains both trend and seasonality dynamics. On the other hand, the seasonal adjustment part of X-13 (the X-11 procedure) can be viewed as a complicated non-parametric model for seasonality which could potentially better capture past seasonality dynamics. It does not offer a way to forecast this seasonality, but you wouldn't need it if you just wanted to forecast the trend.

Also, the example you have chosen (AirPassengers) is a classic example from the Box-Jenkins methodology, and it is included in the X-13 documentation, presumably because it's a series for which it works very well. It's not exactly the most fair test of your suggestion.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.