1
$\begingroup$

I am trying to compare the accuracies of two time series forecasts. I read about the Diebold-Mariano (DM) Test, which tests the null hypothesis of $E[d_t]=0$, where $d_t=g(e_{it})-g(e_{jt})$ is the loss differential between the two forecast errors ($e_t=y_t-\hat{y}_t$, where $y_t$ is the actual and $\hat{y}_t$ is the forecast). One could set $g(e_{it})=e_{it}^2$ or $g(e_{it})=\vert{}e_{it}\vert$ to obtain a measure based on the differential of MSE or MAE respectively.

However, I am wondering whether the result of this test also applies to other accuracy measures which are not based on the forecast error. For instance, consider Mean Directional Accuracy (MDA), defined as:

enter image description here

where $F_t$ is the forecast and $X_t$ is the actual value at time $t$. MDA is thus the proportion of directions correctly predicted by the forecasts.

If a DM test fails to reject $H_0$ (i.e. two competing forecasts have the same precision on average) but the two forecasts have very different MDAs, does this mean that one of the forecasts is more precise (in terms of MDA) or do the conclusions of the DM test also apply to other measures different from MAE and MSE?

$\endgroup$
2
  • $\begingroup$ I think I get this gist of what the question is about, but it would be helpful if all the symbols used were explicitly defined. What is $F_t$, for example? $\endgroup$ Nov 23 '21 at 22:18
  • 1
    $\begingroup$ @Galen You are right, I edited the question accordingly $\endgroup$ Nov 23 '21 at 22:33
1
$\begingroup$

I think Diebold-Mariano test can work on the indicator of directional accuracy instead of the usual absolute error or squared error. If the sample is large enough for the central limit theorem to kick in, you should be able to use the test in the standard way.

Does the superiority of one forecast with respect to one accuracy measure (e.g. absolute error) imply the same forecast's superiority w.r.t. another forecast measure (e.g. squared error or directional accuracy indicator)? No, not in general. Thus you would pick a measure and analyze the different forecasts' performance w.r.t. it separately from other measures, not assuming that the result will be the same for all measures.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.