2
$\begingroup$

I am trying to compare two forecasts using the Mariano Diebold test in R. Both forecasts are for 150 days ahead; that is, on day $t$ I forecast $t+1, t+2, \dotsc, t+150$.

I deduced from this post that my forecast horizon $h=150$. Using that, the Diebold-Mariano test (implemented using function dm.test in "forecast" package in R) gives a p-value of 1 no matter what forecasts I compare.

I looked into the code of this function, and I figured that this is caused by the following 3 lines of code (d is the vector of loss-differential series):

n <- length(d)
k <- ((n + 1 - 2 * h + (h/n) * (h - 1))/n)^(1/2)   
STATISTIC <- STATISTIC * k

Since in my case n = h, the variable k will always be 0 and therefore the test statistic is always zero.

Questions:

  1. Does this mean that we cannot use the Diebold-Mariano test when we are forecasting an entire period at once? I could not find any evidence of this in their paper.
  2. How should I proceed to find a model-based way to compare my forecasts, rather than for example simply taking the MSE?
$\endgroup$

1 Answer 1

2
$\begingroup$

Background
Your forecast horizon is not $150$ but rather a whole vector $h=(h_1,\dotsc,h_{max})=(1,\dotsc,150)$. You only seem to have one observation of forecast errors from the different forecasts for each horizon $h_i$, for $i=1,\dotsc,150$. You cannot compare two forecasts $f_{1,i}$ and $f_{2,i}$ with any confidence because you only have one observation for $f_{1,i}$ and $f_{2,i}$. (Testing equality of means from a sample of size 1 won't work well.)

Answers

  1. You cannot directly use the standard Diebold-Mariano test in this setting.
  2. If you went for the standard Diebold-Mariano setting and collected multiple forecast errors for one horizon (e.g. using rolling windows within the original sample), you could use the standard Diebold-Mariano test.
$\endgroup$
6
  • $\begingroup$ Thank you very much for your quick reply and for editing my question! Your answer explains why I cannot use the Diebold-Mariano test as I was hoping. Concerning point 2: I am not sure if this is the best way to achieve my goal of comparing the performance of 2 forecasts, made at a certain time t, on the entire set of the next 150 days. I expect the overall 'best' forecasting method to use as much information as possible, whereas a rolling horizon would not use all the information in my training set: it ignores the 149 values before. Did I understand this correctly? $\endgroup$
    – Marieke
    Aug 26, 2016 at 11:28
  • 1
    $\begingroup$ As an addition: by the answer to this post, which looks similar to my 'problem', I was lead to think that the Diebold Mariano test compares two forecasts with different horizons by correcting for the fact that larger errors occur a longer period ahead. $\endgroup$
    – Marieke
    Aug 26, 2016 at 11:33
  • 1
    $\begingroup$ @Marieke, You cannot have your cake and eat it too: if you want to do (pseudo) out of sample forecasting you have to estimate models (pseudo) in sample and thus sacrifice some estimation precision w.r.t. the ideal case of estimating in in-sample + out-sample. Regarding the second comment, the idea is interesting and kind of natural. However, the problem is that it is hard to appropriately deal with the increasing error variance along the increasing horizon. How to construct a test statistic that would have good size and power is nontrivial, in my understanding. $\endgroup$ Aug 26, 2016 at 11:38
  • $\begingroup$ I agree. Thank you for showing me the light, I had really gotten stuck in my thoughts. I think I will just stick with the MSE in order to compare the two forecasts. $\endgroup$
    – Marieke
    Aug 26, 2016 at 12:35
  • $\begingroup$ @Marieke, you are welcome! It would be an interesting (but nontrivial) exercise to adapt the Diebold-Mariano test to your setting. Also, note that Diebold himself suggests not using Diebold-Mariano test for comparing models rather than forecasts (Diebold "Comparing predictive accuracy, twenty years later: A personal perspective on the use and abuse of Diebold–Mariano tests", 2015). $\endgroup$ Aug 26, 2016 at 12:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.