I'm using Forecast Principles and Practice 2 to study time series and a doubt came in mind while I was trying to do exercise 7 of chapter 3.

How sensitive are the accuracy measures to the training/test split?

The r code is

retaildata <- readxl::read_excel("retail.xlsx", skip=1)
myts <- ts(retaildata[,"A3349873A"], frequency=12, start=c(1982,4))
x1 <- window(myts, end=c(2010,12))
x2 <- window(myts, start=2011)
f1 <- snaive(x1)

The output is

                    ME     RMSE      MAE       MPE      MAPE
Training set  7.772973 20.24576 15.95676  4.702754  8.109777
Test set     55.300000 71.44309 55.78333 14.900996 15.082019
                 MASE      ACF1 Theil's U
Training set 1.000000 0.7385090        NA
Test set     3.495907 0.5315239  1.297866

The errors are minor in the training set, but is this not expected? I do not know if I understood correctly what the word "sensitive" means in this case.

I thought it would only make sense to compare the accuracy between different forecasting methods and based on test sets.

  • $\begingroup$ It's a good question. My first idea would be to look at other time points where you could split the training and test sample and see whether the test errors differ by much. You could try that. Or ask the FPP2 authors for clarification. $\endgroup$ – Stephan Kolassa Dec 14 '17 at 12:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.