5
$\begingroup$

Suppose I encounter a new forecasting method and I wish to evaluate it against a benchmark. I can obtain forecasts from the two methods and compare them to actual realizations and thus obtain the forecast errors. Applying a loss function on the forecast errors, I can then obtain corresponding forecast losses. I can then test whether the loss differential is different from zero in population by using the Diebold-Mariano test. So far, so good.

Now suppose I encounter multiple new forecasting methods and wish to evaluate the best of them against a benchmark. I do not have advance knowledge w.r.t. which one is the best, but I can find this out empirically using the same test set I will later use for Diebold-Mariano testing. I can

  • obtain forecasts from each of the methods,
  • compare them to actual realizations,
  • obtain the corresponding forecast errors and losses,
  • find the best method (the one with the lowest mean loss) and
  • test whether the loss differential between this method and the benchmark is different from zero using the Diebold-Mariano test.

Well, actually I cannot, because I have selected the best method using the same data as the data I am using to test it against the benchmark. If I use the standard null distribution to compare my test statistic against, I will be likely to reject the null hypothesis more often than the specified significance level – because I have chosen the best performer on this very test set.

What can I do instead? Using a single test set (i.e. without collecting more data), how can I test whether the loss differential between the benchmark and the method that happens to be the best on the test set is different from zero?

$\endgroup$
5
  • $\begingroup$ This could be relevant: sciencedirect.com/science/article/pii/S1574070605010037 $\endgroup$ Sep 22, 2020 at 13:25
  • $\begingroup$ @ChristophHanck, thank you! Do you have in mind anything concrete in that chapter? I think West has mostly researched the unrealistic (and thus practically uninteresting) case of when the pseudo-true values of the models' parameters are known, while I am interested in the realistic case of when they are not. $\endgroup$ Sep 22, 2020 at 13:42
  • $\begingroup$ I was thinking of his discussion of several forecasting models. But I also believe that he shows how the effect of parameter estimation affects evaluation exercises, doesn't he? $\endgroup$ Sep 22, 2020 at 13:59
  • $\begingroup$ @ChristophHanck, I will have to take a closer look. $\endgroup$ Sep 22, 2020 at 14:01
  • 1
    $\begingroup$ How can I determine that a forecast is significantly more accurate than another one? (time series) is related, but not the same, because you are asking for a comparison between (the best one out of) a lot of forecasts and one specific benchmark, whereas the MCB and the Nemenyi test I recommend in that thread compare all candidates either to the best one or the mean, or a random ranking of all methods. $\endgroup$ Mar 25, 2021 at 7:59

1 Answer 1

2
$\begingroup$

An answer to a very similar (but not the actual $\color{red}{^*}$) question is, White's Reality Check and Hansen's Superior Predictive Ability (SPA) Test. See Section 17.5.2 in Elliott & Timmermann (2016) for a summary of both tests. Below I give a summary of the summary (consisting in no small part of direct quotes):

White (2000) asks how confident we can be that the best forecast, among a set of competing forecasts, is genuinely better than a prespecified benchmark, given that the best forecast is selected from a potentially large set of models. The null hypothesis tested by the Reality Check is that the benchmark model is not inferior to any of the $m$ alternatives: $$ H_0\colon \max_{k=1,\dots,m}\mathbb{E}[d_{k,t+1|t}(\beta^∗_k)]\leq 0, $$ whereas the alternative is that at least one model produces lower expected loss than the benchmark: $$ H_1\colon \max_{k=1,\dots,m}\mathbb{E}[d_{k,t+1|t}(\beta^∗_k)]>0. $$ Here, $d_{k,t+1|t}(\beta^∗_k)$ is the loss differential between a prespecified benchmark model and its $k$th competitor, and $\beta^∗_k=\text{plim}(\hat\beta_{k,T})$ is the pseudo-true value (given the estimation method) of the parameter vector $\beta_k$ of the $k$th model.

Hansen (2005) notes that the bootstrap procedure used for constructing the null distribution of the test statistic in White (2000) means that, in practice, the assumption under the null hypothesis is that $\mathbb{E}[d_k(\beta^∗)]=0$. He shows that if the maximum of $\mathbb{E}(\bar{d})$ is negative, then with probability 1 we get a degenerate distribution of the test statistic when the benchmark model is better than all other models. Here, $\bar{d}$ is the vector of sample averages of loss differentials. Hansen's Superior Predictive Ability (SPA) Test modifies the Reality Check by normalizing and recentering the test statistic to get rid of the assumption mentioned above.


$\color{red}{^*}$ An important caveat: the interest in White's Reality Check and Hansen's Superior Predictive Ability (SPA) Test is in comparing forecasting performance of competing models in a hypothetical setting when the pseudo-true values of the models' parameters are known. In other words, their null hypotheses assume perfect estimation precision of the underlying forecasting models. This is not necessarily an interesting question in practice, where true values are not known but their imperfect estimates are used. Importantly, this assumption is at odds with the assumptions of the Diebold-Mariano test, but is in line with some papers by Clark, McCracken and West. Therefore, this answer addresses a slightly different question that the OP is asking.

A direct answer to the OPs question is still sought after.


References

  1. Elliott, G. & Timmermann, A. (2016). Economic Forecasting. Princeton University Press.
  2. Hansen, P. R. (2005). A test for superior predictive ability. Journal of Business & Economic Statistics, 23(4), 365-380.
  3. White, H. (2000). A reality check for data snooping. Econometrica, 68(5), 1097-1126.
$\endgroup$
3
  • $\begingroup$ How do I get $\color{red}*$ to be a superscript? $\color{red}^*$ does not work. $\endgroup$ Nov 14, 2019 at 11:45
  • $\begingroup$ It think $\color{red}{^*}$works$\color{red}{^*}$? $\endgroup$ Nov 14, 2019 at 20:54
  • 1
    $\begingroup$ @COOLSerdash, brilliant! That works indeed. $\endgroup$ Nov 14, 2019 at 21:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.