Suppose I encounter a new forecasting method and I wish to evaluate it against a benchmark. I can obtain forecasts from the two methods and compare them to actual realizations and thus obtain the forecast errors. Applying a loss function on the forecast errors, I can then obtain corresponding forecast losses. I can then test whether the loss differential is different from zero in population by using the Diebold-Mariano test. So far, so good.
Now suppose I encounter multiple new forecasting methods and wish to evaluate the best of them against a benchmark. I do not have advance knowledge w.r.t. which one is the best, but I can find this out empirically using the same test set I will later use for Diebold-Mariano testing. I can
- obtain forecasts from each of the methods,
- compare them to actual realizations,
- obtain the corresponding forecast errors and losses,
- find the best method (the one with the lowest mean loss) and
- test whether the loss differential between this method and the benchmark is different from zero using the Diebold-Mariano test.
Well, actually I cannot, because I have selected the best method using the same data as the data I am using to test it against the benchmark. If I use the standard null distribution to compare my test statistic against, I will be likely to reject the null hypothesis more often than the specified significance level – because I have chosen the best performer on this very test set.
What can I do instead? Using a single test set (i.e. without collecting more data), how can I test whether the loss differential between the benchmark and the method that happens to be the best on the test set is different from zero?