9
$\begingroup$

Reproducible Example

Look at this reproducible example:

  • I have a time series that I want to forecast. For the sake of reproducibility, I'll just take AirPassanger.
  • Let's say that I tried to fit 5 models: ETS, (auto)ARIMA, drifted SNAIVE, NNETAR, SNAIVE.
  • I kept aside part of the data as test set and I trained all my models on the training set.
  • I calculated the accuracy on the test set for each model.

library(forecast)
#> Registered S3 method overwritten by 'quantmod':
#>   method            from
#>   as.zoo.data.frame zoo

# train and test
ts <- AirPassengers
trn <- window(ts, end = c(1958, 12))
tst <- window(ts, start = c(1959, 1), end = c(1960, 12))

# models
set.seed(1)
mdl_ets <- ets(trn, lambda = 0)
mdl_arm <- auto.arima(trn, lambda = 0)
mdl_lag <- forecast:::lagwalk(trn, lag = 12, drift = TRUE, lambda = 0, biasadj = TRUE)
mdl_nnt <- nnetar(trn, h = length(tst))
mdl_snv <- snaive(trn, h = length(tst))

# forecast
frc_ets <- forecast(mdl_ets, h = length(tst))
frc_arm <- forecast(mdl_arm, h = length(tst))
frc_lag <- forecast(mdl_lag, h = length(tst), lambda = mdl_lag$lambda)
frc_nnt <- forecast(mdl_nnt, h = length(tst))
frc_snv <- forecast(mdl_snv, h = length(tst))

# plot
plot(ts)
lines(frc_ets$mean, col = "red")
lines(frc_arm$mean, col = "blue")
lines(frc_lag$mean, col = "green")
lines(frc_nnt$mean, col = "violet")
lines(frc_snv$mean, col = "orange")



# accuracy on test set
rbind(
        ets = accuracy(frc_ets, tst)[2,],
        arm = accuracy(frc_arm, tst)[2,],
        lag = accuracy(frc_lag, tst)[2,],
        nnt = accuracy(frc_nnt, tst)[2,],
        snv = accuracy(frc_snv, tst)[2,]
)
#>            ME     RMSE      MAE       MPE      MAPE      MASE       ACF1 Theil's U
#> ets 17.315122 26.53712 21.41598  3.452246  4.468821 0.7494900 0.42253991 0.5044923
#> arm 39.447258 43.18367 39.44726  8.516316  8.516316 1.3805262 0.46359970 0.8430396
#> lag -5.831543 17.01594 13.27878 -1.241407  3.005621 0.4647144 0.02788362 0.3432294
#> nnt 22.854463 30.06140 24.64835  4.661514  5.166691 0.8626125 0.45103130 0.5832227
#> snv 71.250000 76.99459 71.25000 15.523355 15.523355 2.4935191 0.72846283 1.5197525

Created on 2021-03-25 by the reprex package (v0.3.0)

Question

I can see from the graph and the accuracy table that the drifted SNAIVE (that I called lag) is probably the best.

Is there a way I can determine that it is significantly the best?

What I tried

I was looking for some tests of significance. I found the Diebold-Mariano Test (package forecast and package multDM). However, I think the test is specific for cross-validated one-step ahead forecasts, which is not my case.

More info

In my specific problem, I'm not the guy that builds the model. I have the original data and I only receive a forecast corresponding to my test set from an external source. I don't have the parameters of the model, nor the confidence intervals.

If you need a definition of "accuracy" from me, just assume that I will probably look at MAPE or RMSE, but only because they are easier to communicate to non-technical people. I'm aware of the limits of MAPE.

Can someone help me?

Also, this is my first question here, so please let me know if I'm correctly following all the usual standards.

UPDATE

Based on @Stephan Kolassa's answer, I suppose this is the right code.

library(tsutils)

abs_err <- abs(cbind(
  ets = as.numeric(frc_ets$mean),
  arm = as.numeric(frc_arm$mean),
  lag = as.numeric(frc_lag$mean),
  nnt = as.numeric(frc_nnt$mean),
  snv = as.numeric(frc_snv$mean)
) - as.numeric(tst))

nemenyi(abs_err, plottype="vmcb")
#> Friedman and Nemenyi Tests
#> The confidence level is 5%
#> Number of observations is 24 and number of methods is 5
#> Friedman test p-value: 0.0000 - Ha: Different
#> Critical distance: 1.2451

enter image description here

tsutils::nemenyi(abs_err, plottype="vline")

enter image description here

tsutils::nemenyi(abs_err, plottype="matrix")

enter image description here

Based on this I can't say that lag is significantly more accurate than nnt and ets. However, there is enough evidence to say that arm and snv are significantly less accurate. (Correct?)

A couple of points:

  • I've tried to input different kind of errors: absolute errors, squared errors, absolute percentage errors. There is no difference.
  • TStools, the suggested R package, is available only as GitHub version. tsutils is the correspondent CRAN version.
  • There is another package that performs those tests: PMCMR, but it doesn't provide the same intuitive visualizations.
  • Friedman test is also included in the stats package
friedman.test(abs_err)
#> 
#>  Friedman rank sum test
#> 
#> data:  abs_err
#> Friedman chi-squared = 61.167, df = 4, p-value = 1.649e-12

PMCMR::posthoc.friedman.nemenyi.test(abs_err)
#> 
#>  Pairwise comparisons using Nemenyi multiple comparison test 
#>              with q approximation for unreplicated blocked data
#> 
#> data: abs_err
#> 
#>     ets     arm     lag     nnt    
#> arm 0.02875 -       -       -      
#> lag 0.85358 0.00082 -       -      
#> nnt 0.99998 0.03754 0.80900 -      
#> snv 8.8e-08 0.03754 1.4e-10 1.5e-07
#> 
#> P value adjustment method: none
Improve this question
$\endgroup$
1
  • 4
    $\begingroup$ Thank you for a very nice first post. I wish all first posts were of this quality... $\endgroup$
    – Stephan Kolassa
    Commented Mar 25, 2021 at 6:52

2 Answers 2

Reset to default
4
$\begingroup$

Forecasters (those who do worry about statistical significance, which is still not all of us; compare Diebold's 2015 recollection of "bewilderment as to why anyone would care about the subject" in the referee report to their initial submission) will often happily summarize multiple steps ahead to obtain a mean error per series and method across time, then compare these summaries using the Diebold-Mariano test, even if it is in principle only intended to compare a single time step. However, the issue why the DM test is not very helpful is that it compares only two forecasts, and you have many, so you have a multiple comparisons problem.

In such a case, the standard approach is the "multiple comparisons to the best" (MCB) test originally proposed by Koning et al. (2005) for a re-analysis of the M3 forecasting competition. Most recently it has been applied to submissions in the M5 forecasting competition as well. It is rank-based, so it works with any accuracy measure (and appropriate point forecasts, Kolassa, 2020, SCNR). A related alternative would be the Friedman-Nemenyi test (Demsar, 2006).

Both the MCB and the Nememyi test are implemented in the TStools package for R. An empirical comparison between the two is given by Hibon et al.'s 2012 ISF presentation.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Very interesting! I wonder why I have not seen these names before. The forecasting textbooks (Elliott & Timmermann's, Diebold's, FPP and more) and papers I have read mention Diebold-Mariano, Giacomini-White, some of Clark, McCracken or West, then White's Reality Check and Hansen et al.'s Model Confidence Set. Why have the names you mention gone under the radar? $\endgroup$
    – Richard Hardy
    Commented Mar 25, 2021 at 7:12
  • $\begingroup$ @RichardHardy: that's a very good question, which I sadly don't have a good answer to. There are a few forecasters who push these tests (e.g., Kourentzes, Petropoulos et al.), who also influenced Makridakis so the MCB test is used in the M5 competition, which may help its popularity. But the majority of presentations I see at the ISF don't use this kind of test and simply report accuracies without testing for statistical significance. ... $\endgroup$
    – Stephan Kolassa
    Commented Mar 25, 2021 at 7:38
  • $\begingroup$ ... As to textbooks, neither FPP3 nor Principles of Business Forecasting (with Kourentzes as an author on the 2nd edition), nor our textbook mention it. It may well be that people find the entire concept too confusing, with the three dimensions you need to keep in mind (series, methods, forecasting periods), possibly for multiple error measures as well. I would say that more marketing is needed. Which is why I am so glad about a well-posed question on this topic here at CV. $\endgroup$
    – Stephan Kolassa
    Commented Mar 25, 2021 at 7:40
  • 1
    $\begingroup$ Once I get acquainted with the tests, I might join in advertising them where applicable and appropriate. I am glad I found them mentioned in your answer! Perhaps they may even shed light on this earlier question of mine. $\endgroup$
    – Richard Hardy
    Commented Mar 25, 2021 at 7:51
  • $\begingroup$ @StephanKolassa thank you very much for your great answer! I've made an update to my question. I've tried the tests and packages you suggested. I thought it could be useful for other users to have a fully functional code. If you think I should include it in your answer, let me know (provided my code and my insights are correct in your opinion..!). $\endgroup$
    – Edo
    Commented Mar 25, 2021 at 16:39
0
$\begingroup$

Best can be defined many ways. More accurate is the way I define it as a practitioner not a researcher. I just compare the predicted results to the actual results with a MAPE. But even for accuracy there are many alternatives.

There is of course no test of statistical significance this ways. Since the results are real I don't really think that matters. No sample and population issues are involved.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.