I'm using Forecast Principles and Practice 2 to study time series and a doubt came in mind while I was trying to do exercise 7 of chapter 3.
How sensitive are the accuracy measures to the training/test split?
The r code is
retaildata <- readxl::read_excel("retail.xlsx", skip=1) myts <- ts(retaildata[,"A3349873A"], frequency=12, start=c(1982,4)) x1 <- window(myts, end=c(2010,12)) x2 <- window(myts, start=2011) autoplot(cbind(x1,x2)) f1 <- snaive(x1) accuracy(f1,x2) checkresiduals(f1)
The output is
ME RMSE MAE MPE MAPE Training set 7.772973 20.24576 15.95676 4.702754 8.109777 Test set 55.300000 71.44309 55.78333 14.900996 15.082019 MASE ACF1 Theil's U Training set 1.000000 0.7385090 NA Test set 3.495907 0.5315239 1.297866
The errors are minor in the training set, but is this not expected? I do not know if I understood correctly what the word "sensitive" means in this case.
I thought it would only make sense to compare the accuracy between different forecasting methods and based on test sets.