Look at this reproducible example:
- I have a time series that I want to forecast. For the sake of reproducibility, I'll just take
- Let's say that I tried to fit 5 models: ETS, (auto)ARIMA, drifted SNAIVE, NNETAR, SNAIVE.
- I kept aside part of the data as test set and I trained all my models on the training set.
- I calculated the accuracy on the test set for each model.
library(forecast) #> Registered S3 method overwritten by 'quantmod': #> method from #> as.zoo.data.frame zoo # train and test ts <- AirPassengers trn <- window(ts, end = c(1958, 12)) tst <- window(ts, start = c(1959, 1), end = c(1960, 12)) # models set.seed(1) mdl_ets <- ets(trn, lambda = 0) mdl_arm <- auto.arima(trn, lambda = 0) mdl_lag <- forecast:::lagwalk(trn, lag = 12, drift = TRUE, lambda = 0, biasadj = TRUE) mdl_nnt <- nnetar(trn, h = length(tst)) mdl_snv <- snaive(trn, h = length(tst)) # forecast frc_ets <- forecast(mdl_ets, h = length(tst)) frc_arm <- forecast(mdl_arm, h = length(tst)) frc_lag <- forecast(mdl_lag, h = length(tst), lambda = mdl_lag$lambda) frc_nnt <- forecast(mdl_nnt, h = length(tst)) frc_snv <- forecast(mdl_snv, h = length(tst)) # plot plot(ts) lines(frc_ets$mean, col = "red") lines(frc_arm$mean, col = "blue") lines(frc_lag$mean, col = "green") lines(frc_nnt$mean, col = "violet") lines(frc_snv$mean, col = "orange")
# accuracy on test set rbind( ets = accuracy(frc_ets, tst)[2,], arm = accuracy(frc_arm, tst)[2,], lag = accuracy(frc_lag, tst)[2,], nnt = accuracy(frc_nnt, tst)[2,], snv = accuracy(frc_snv, tst)[2,] ) #> ME RMSE MAE MPE MAPE MASE ACF1 Theil's U #> ets 17.315122 26.53712 21.41598 3.452246 4.468821 0.7494900 0.42253991 0.5044923 #> arm 39.447258 43.18367 39.44726 8.516316 8.516316 1.3805262 0.46359970 0.8430396 #> lag -5.831543 17.01594 13.27878 -1.241407 3.005621 0.4647144 0.02788362 0.3432294 #> nnt 22.854463 30.06140 24.64835 4.661514 5.166691 0.8626125 0.45103130 0.5832227 #> snv 71.250000 76.99459 71.25000 15.523355 15.523355 2.4935191 0.72846283 1.5197525
Created on 2021-03-25 by the reprex package (v0.3.0)
I can see from the graph and the accuracy table that the drifted SNAIVE (that I called
lag) is probably the best.
Is there a way I can determine that it is significantly the best?
What I tried
I was looking for some tests of significance. I found the Diebold-Mariano Test (package
forecast and package
multDM). However, I think the test is specific for cross-validated one-step ahead forecasts, which is not my case.
In my specific problem, I'm not the guy that builds the model. I have the original data and I only receive a forecast corresponding to my test set from an external source. I don't have the parameters of the model, nor the confidence intervals.
If you need a definition of "accuracy" from me, just assume that I will probably look at MAPE or RMSE, but only because they are easier to communicate to non-technical people. I'm aware of the limits of MAPE.
Can someone help me?
Also, this is my first question here, so please let me know if I'm correctly following all the usual standards.
Based on @Stephan Kolassa's answer, I suppose this is the right code.
library(tsutils) abs_err <- abs(cbind( ets = as.numeric(frc_ets$mean), arm = as.numeric(frc_arm$mean), lag = as.numeric(frc_lag$mean), nnt = as.numeric(frc_nnt$mean), snv = as.numeric(frc_snv$mean) ) - as.numeric(tst)) nemenyi(abs_err, plottype="vmcb") #> Friedman and Nemenyi Tests #> The confidence level is 5% #> Number of observations is 24 and number of methods is 5 #> Friedman test p-value: 0.0000 - Ha: Different #> Critical distance: 1.2451
Based on this I can't say that
lag is significantly more accurate than
ets. However, there is enough evidence to say that
snv are significantly less accurate. (Correct?)
A couple of points:
- I've tried to input different kind of errors: absolute errors, squared errors, absolute percentage errors. There is no difference.
TStools, the suggested R package, is available only as GitHub version.
tsutilsis the correspondent CRAN version.
- There is another package that performs those tests:
PMCMR, but it doesn't provide the same intuitive visualizations.
- Friedman test is also included in the
friedman.test(abs_err) #> #> Friedman rank sum test #> #> data: abs_err #> Friedman chi-squared = 61.167, df = 4, p-value = 1.649e-12 PMCMR::posthoc.friedman.nemenyi.test(abs_err) #> #> Pairwise comparisons using Nemenyi multiple comparison test #> with q approximation for unreplicated blocked data #> #> data: abs_err #> #> ets arm lag nnt #> arm 0.02875 - - - #> lag 0.85358 0.00082 - - #> nnt 0.99998 0.03754 0.80900 - #> snv 8.8e-08 0.03754 1.4e-10 1.5e-07 #> #> P value adjustment method: none