Alternatives for Diebold-Mariano test when comparing the best forecast among many against a benchmark

Question

Suppose I encounter a new forecasting method and I wish to evaluate it against a benchmark. I can obtain forecasts from the two methods and compare them to actual realizations and thus obtain the forecast errors. Applying a loss function on the forecast errors, I can then obtain corresponding forecast losses. I can then test whether the loss differential is different from zero in population by using the Diebold-Mariano test. So far, so good.

Now suppose I encounter multiple new forecasting methods and wish to evaluate the best of them against a benchmark. I do not have advance knowledge w.r.t. which one is the best, but I can find this out empirically using the same test set I will later use for Diebold-Mariano testing. I can

• obtain forecasts from each of the methods,
• compare them to actual realizations,
• obtain the corresponding forecast errors and losses,
• find the best method (the one with the lowest mean loss) and
• test whether the loss differential between this method and the benchmark is different from zero using the Diebold-Mariano test.

Well, actually I cannot, because I have selected the best method using the same data as the data I am using to test it against the benchmark. If I use the standard null distribution to compare my test statistic against, I will be likely to reject the null hypothesis more often than the specified significance level – because I have chosen the best performer on this very test set.

What can I do instead? Using a single test set (i.e. without collecting more data), how can I test whether the loss differential between the benchmark and the method that happens to be the best on the test set is different from zero?

Answer 1

An answer to a very similar (but not the actual $\color{red}{^*}$) question is, White's Reality Check and Hansen's Superior Predictive Ability (SPA) Test. See Section 17.5.2 in Elliott & Timmermann (2016) for a summary of both tests. Below I give a summary of the summary (consisting in no small part of direct quotes):

White (2000) asks how confident we can be that the best forecast, among a set of competing forecasts, is genuinely better than a prespecified benchmark, given that the best forecast is selected from a potentially large set of models. The null hypothesis tested by the Reality Check is that the benchmark model is not inferior to any of the $m$ alternatives: $$ H_0\colon \max_{k=1,\dots,m}\mathbb{E}[d_{k,t+1|t}(\beta^∗_k)]\leq 0, $$ whereas the alternative is that at least one model produces lower expected loss than the benchmark: $$ H_1\colon \max_{k=1,\dots,m}\mathbb{E}[d_{k,t+1|t}(\beta^∗_k)]>0. $$ Here, $d_{k,t+1|t}(\beta^∗_k)$ is the loss differential between a prespecified benchmark model and its $k$th competitor, and $\beta^∗_k=\text{plim}(\hat\beta_{k,T})$ is the pseudo-true value (given the estimation method) of the parameter vector $\beta_k$ of the $k$th model.

Hansen (2005) notes that the bootstrap procedure used for constructing the null distribution of the test statistic in White (2000) means that, in practice, the assumption under the null hypothesis is that $\mathbb{E}[d_k(\beta^∗)]=0$. He shows that if the maximum of $\mathbb{E}(\bar{d})$ is negative, then with probability 1 we get a degenerate distribution of the test statistic when the benchmark model is better than all other models. Here, $\bar{d}$ is the vector of sample averages of loss differentials. Hansen's Superior Predictive Ability (SPA) Test modifies the Reality Check by normalizing and recentering the test statistic to get rid of the assumption mentioned above.

---

$\color{red}{^*}$ An important caveat: the interest in White's Reality Check and Hansen's Superior Predictive Ability (SPA) Test is in comparing forecasting performance of competing models in a hypothetical setting when the pseudo-true values of the models' parameters are known. In other words, their null hypotheses assume perfect estimation precision of the underlying forecasting models. This is not necessarily an interesting question in practice, where true values are not known but their imperfect estimates are used. Importantly, this assumption is at odds with the assumptions of the Diebold-Mariano test, but is in line with some papers by Clark, McCracken and West. Therefore, this answer addresses a slightly different question that the OP is asking.

A direct answer to the OPs question is still sought after.

---

References

1. Elliott, G. & Timmermann, A. (2016). Economic Forecasting. Princeton University Press.
2. Hansen, P. R. (2005). A test for superior predictive ability. Journal of Business & Economic Statistics, 23(4), 365-380.
3. White, H. (2000). A reality check for data snooping. Econometrica, 68(5), 1097-1126.