On a problem with the implementation of the test of Diebold and Mariano for equal predictive accuracy

Question

To recall how the test of Diebold and Mariano is set up I utilize the parsimonious words of Diebold himself:

Now, I have been trying to perform this test on the respective vector of errors of two competing forecasts using R but something I do not understand happens. Here is the output:

After performing the test once with the two vectors of errors and obtaining as p-value 5.287e^(-11) I wanted to check if everything was working correctly and I multiplied the first vector of errors by 10. Performing the test a second time the p-value is lower than in the first test, how is this possible?

The null hypothesis is that both forecast have the same predictive ability, so increasing the difference between the errors it seems to me that the pvalue should decrease. What have I missed?

Initially I thought it was a code mistake but the same thing happens when I perform the test with an other function in Matlab.

EDIT: After various tries It now seems that multiplying one of the two vectors of errors by a power of 10 does not change the p-value of the test. Is this by design? I am starting to think that in $\frac{\bar{d}_{12}}{\hat{\sigma}_{d_{12}}}$ both the terms get multiplied by 10 so the p-value does not change Except that the numerator does not get multiplied exactly by 10.

Also it still does not make sense to me how after increasing the values of one error set the p-value remains the same, surely the model that produced the untouched set with smaller errors should be the better model.

If you want to test my claim you can use the datasets in this question and try multiplying one by a power of ten and redo the test.

Answer 1

First, the data used in the linked post are squared forecast errors rather than the original forecast errors. If that is not noticed and the squared forecast errors are used in place of the original errors, you may already expect some weird behaviour. But let us work with what we have and hope that the cause of the trouble lies elsewhere (which I will show to be the case).

Second, the first series is about two orders of magnitude larger than the second series. Therefore, the error difference $\bar{d}_{12}$ is essentially equal to the first error $e_1$ alone, $\bar{d}_{12} = e_1-e_2 \approx e_1$. Then also the standard deviation of the difference $\bar{d}_{12}$ is close to the standard deviation of the first error $e_1$ alone, $\hat{\sigma}_{12} \approx \hat{\sigma}_1$. Thus

$$\text{DM} = \frac{\bar{d}_{12}}{\hat{\sigma}_{12}} \approx \frac{\bar{e}_1}{\hat{\sigma}_1}.$$

Then you would actually anticipate the behaviour you are observing: multiplying the first series by 10 yields almost no change in the $\text{DM}$ statistic since the mean as well as the standard deviation react proportionally to scaling of the original variable: $\text{E}(10 \cdot x)=10 \cdot \text{E}(x)$ and $\sigma(10 \cdot x)=10 \cdot \sigma(x)$ (the same also holds for their sample counterparts). Essentially, both the numerator and the denominator of $\text{DM}$ get multiplied by 10, and the fraction does not change.

On the other hand, if you try the dm.test function with "normal" data, you will get sensible results. For example, try

library(forecast)
n=100 # sample size
set.seed(1); e1=rnorm(n) # generate a vector of standard normal errors
set.seed(3); e2=rnorm(n) # generate a second vector of standard normal errors
e3=e1*10 # let the third vector of errors equal the first vector scaled by 10
dm.test(e1,e2,alternative="two.sided",h=1,power=1)
dm.test(e1,e2,alternative="two.sided",h=1,power=2)
dm.test(e3,e2,alternative="two.sided",h=1,power=1)
dm.test(e3,e2,alternative="two.sided",h=1,power=2)

Clearly, the original error series are not too different in the Diebold-Mariano sense and the $p$-value of the test is high. However, having multiplied one of the error series by 10 a clear difference in the Diebold-Mariano sense emerges with a very low $p$-value of the test.

Edit: To respond to some of the comments, here is my take on the intuition for why multiplying $e_1$ by 10 does not push the $\text{DM}$ statistic away from zero (and this way does not increase its statistical significance). As we have seen from the formula above in this post, when $e_1$ is already way larger than $e_2$ the $\text{DM}$ statistic behaves as the ratio of the mean of $e_1$ to the standard deviation of $e_1$. Multiplying $e_1$ by 10 increases the mean and the standard deviation proportionally. Although the error becomes larger or average, the uncertainty about its size increases proportionally. Thus we are as uncertain as we previously were whether $e_1$ is actually mean zero or not. As a counterexample, suppose we increase only the magnitude of the error but not its standard deviation by adding a vector of constants to $e_1$ rather than multiplying $e_1$ by 10. Then we would become more certain that $e_1$ is not mean-zero and the $\text{DM}$ statistic would move away from zero and become more statistically significant.