1
$\begingroup$

To recall how the test of Diebold and Mariano is set up I utilize the parsimonious words of Diebold himself:

enter image description here

enter image description here

Now, I have been trying to perform this test on the respective vector of errors of two competing forecasts using R but something I do not understand happens. Here is the output:

enter image description here

After performing the test once with the two vectors of errors and obtaining as p-value 5.287e^(-11) I wanted to check if everything was working correctly and I multiplied the first vector of errors by 10. Performing the test a second time the p-value is lower than in the first test, how is this possible?

The null hypothesis is that both forecast have the same predictive ability, so increasing the difference between the errors it seems to me that the pvalue should decrease. What have I missed?

Initially I thought it was a code mistake but the same thing happens when I perform the test with an other function in Matlab.

EDIT: After various tries It now seems that multiplying one of the two vectors of errors by a power of 10 does not change the p-value of the test. Is this by design? I am starting to think that in $\frac{\bar{d}_{12}}{\hat{\sigma}_{d_{12}}}$ both the terms get multiplied by 10 so the p-value does not change Except that the numerator does not get multiplied exactly by 10.

Also it still does not make sense to me how after increasing the values of one error set the p-value remains the same, surely the model that produced the untouched set with smaller errors should be the better model.

If you want to test my claim you can use the datasets in this question and try multiplying one by a power of ten and redo the test.

$\endgroup$
3
+50
$\begingroup$

First, the data used in the linked post are squared forecast errors rather than the original forecast errors. If that is not noticed and the squared forecast errors are used in place of the original errors, you may already expect some weird behaviour. But let us work with what we have and hope that the cause of the trouble lies elsewhere (which I will show to be the case).

Second, the first series is about two orders of magnitude larger than the second series. Therefore, the error difference $\bar{d}_{12}$ is essentially equal to the first error $e_1$ alone, $\bar{d}_{12} = e_1-e_2 \approx e_1$. Then also the standard deviation of the difference $\bar{d}_{12}$ is close to the standard deviation of the first error $e_1$ alone, $\hat{\sigma}_{12} \approx \hat{\sigma}_1$. Thus

$$\text{DM} = \frac{\bar{d}_{12}}{\hat{\sigma}_{12}} \approx \frac{\bar{e}_1}{\hat{\sigma}_1}.$$

Then you would actually anticipate the behaviour you are observing: multiplying the first series by 10 yields almost no change in the $\text{DM}$ statistic since the mean as well as the standard deviation react proportionally to scaling of the original variable: $\text{E}(10 \cdot x)=10 \cdot \text{E}(x)$ and $\sigma(10 \cdot x)=10 \cdot \sigma(x)$ (the same also holds for their sample counterparts). Essentially, both the numerator and the denominator of $\text{DM}$ get multiplied by 10, and the fraction does not change.

On the other hand, if you try the dm.test function with "normal" data, you will get sensible results. For example, try

library(forecast)
n=100 # sample size
set.seed(1); e1=rnorm(n) # generate a vector of standard normal errors
set.seed(3); e2=rnorm(n) # generate a second vector of standard normal errors
e3=e1*10 # let the third vector of errors equal the first vector scaled by 10
dm.test(e1,e2,alternative="two.sided",h=1,power=1)
dm.test(e1,e2,alternative="two.sided",h=1,power=2)
dm.test(e3,e2,alternative="two.sided",h=1,power=1)
dm.test(e3,e2,alternative="two.sided",h=1,power=2)

Clearly, the original error series are not too different in the Diebold-Mariano sense and the $p$-value of the test is high. However, having multiplied one of the error series by 10 a clear difference in the Diebold-Mariano sense emerges with a very low $p$-value of the test.

Edit: To respond to some of the comments, here is my take on the intuition for why multiplying $e_1$ by 10 does not push the $\text{DM}$ statistic away from zero (and this way does not increase its statistical significance). As we have seen from the formula above in this post, when $e_1$ is already way larger than $e_2$ the $\text{DM}$ statistic behaves as the ratio of the mean of $e_1$ to the standard deviation of $e_1$. Multiplying $e_1$ by 10 increases the mean and the standard deviation proportionally. Although the error becomes larger or average, the uncertainty about its size increases proportionally. Thus we are as uncertain as we previously were whether $e_1$ is actually mean zero or not. As a counterexample, suppose we increase only the magnitude of the error but not its standard deviation by adding a vector of constants to $e_1$ rather than multiplying $e_1$ by 10. Then we would become more certain that $e_1$ is not mean-zero and the $\text{DM}$ statistic would move away from zero and become more statistically significant.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you very much Richard for your answer, you are absolutely right about the data. Your explanation for why the p-valued does not change was something I shortly thought about in my edit, except that performing a test very similar to the one you did I get your same result, i.e., the p-value DOES change with "normal" data. So now I am thinking that the problem lies in the data and it's not the fractions "fault". $\endgroup$ – Monolite Sep 17 '15 at 16:45
  • $\begingroup$ Searching for an answer I stumbled upon these words by clark "the root of the problem is that, at the population level, if the null hypothesis is that the smaller model is the true one, the forecast errors from the competing models are exactly the same and perfectly correlated, which means that the numerator and denominator of a Diebold-Mariano test are each limiting to zero as the estimation sample and prediction sample grow." I don't understand why perfect correlation would be a problem because the CLT for stationary processes that is being used should work even with perfect correlation. $\endgroup$ – Monolite Sep 17 '15 at 16:46
  • $\begingroup$ I am assuming that the theorem Diebold and Mariano use is a standard CLT for stationary processes: Let $$Y_t = \mu + \sum_i \theta_j \epsilon_{t-j}$$ where $\epsilon_t$ is a sequence of IID random variables with $E(\epsilon_t^2)< \infty$ and $\sum_j |\theta_j| < \infty$. Then $$\sqrt{T}(\bar{Y}_t - \mu) \rightarrow N(0, \sum_j \gamma_j)$$. So maybe dependence is the cause of the bad results, but I think the independence assumption could be relaxed. If you have any ideas or you think that the fraction problem is still the cause feel very free to correct me/instruct me. $\endgroup$ – Monolite Sep 17 '15 at 17:02
  • 1
    $\begingroup$ The root cause is that $e_1$ was way (about a hundred times) larger than $e_2$ at the start. Then multiplying by 10 did not have much effect: $e_1$ remained way larger. However, when you have errors are that closer to each other in magnitude, you will notice a difference when multiplying one by 10. $\endgroup$ – Richard Hardy Sep 18 '15 at 5:20
  • 1
    $\begingroup$ This could certainly be the cause of why the p-value does not change or changes by very little, but the fact that this happens for p-values that are not in a significant rejection of the null range (>0.05) still seems very strange behaviour to me. The fact that no matter how much I distance the errors I can't seem to push the p-value in a rejection region seems wrong to me. $\endgroup$ – Monolite Sep 18 '15 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.