3
$\begingroup$

Perhaps I'm missing a simple conceptual point here. But do the error statistics (RMSE, MAE) not tell which is the best forecast by presenting the lowest figures between the forecast and the actual figures? Why then do we need to compare two forecasts directly, given we've already established which is one is superior via RMSE etc? Isn't our interest in establishing which one replicates values closest to the actual figures?

Secondly how would one go about interpreting the following Diebold Mariano test I've conducted. Would the below be correct?


<>prob

H0: Both forecasts have the same accuracy
H1: The forecasts do not have the same accuracy

$<0.05$ reject null, accept alternative hypothesis
$>0.05$ accept null, both forecasts have the same accuracy


>Prob

H0: Both forecasts have the same accuracy
H1: Forecast 1 is less accurate than forecast 2

$<0.05$ reject null, accept alternative hypothesis
$>0.05$ accept null, Both Forecasts have the same accuracy


<.Prob

H0: Both forecasts have the same accuracy
H1: Forecast 2 is more accurate than Forecast 1

$<0.05$ reject null, accept alternative hypothesis
$>0.05$ accept null, Both Forecasts have the same accuracy


enter image description here

This can then ultimately be written as something close to the following: The table shows the Diebold-Mariano test statistics for different sub-periods. It is employed to test whether the RMSE performance between the Short Moving Average of 5 days and other volatility forecasting models are statistically different from one another. A negative value indicates that the Short Moving Average of 5 days is better than the second model.

i.e. Negative values of the Diebold–Mariano test show that the squared errors of the model listed first (short moving average) are lower than those of the model listed last.

$\endgroup$
  • 2
    $\begingroup$ The Diebold-Mariano test is similar to a $t$-test in a linear regression model: you see that a coefficient is nonzero, but you want to inspect how likely it is that the deviation from zero is due to randomness. If the $p$-value corresponding to the $t$-test is low, the estimated difference from zero is unlikely to be as large as it is due to pure randomness. Similarly, the difference between RMSEs is nonzero, but you want to check whether this is due to randomness. Then you use a $t$-test as proposed by Diebold & Mariano. $\endgroup$ – Richard Hardy Apr 11 '17 at 11:42
  • $\begingroup$ I have posted a full answer now. $\endgroup$ – Richard Hardy Jun 24 '17 at 19:46
2
$\begingroup$

Why then do we need to compare two forecasts directly, given we've already established which is one is superior via RMSE etc? Isn't our interest in establishing which one replicates values closest to the actual figures?

Let Diebold himself answer this for you (Diebold, 2015, second paragraph in the Introduction):

The need for formal tests for comparing predictive accuracy is surely obvious. We’ve all seen hundreds of predictive horse races, with one or the other declared the “winner” (usually the new horse in the stable), but with no consideration given to the statistical significance of the victory. Such predictive comparisons are incomplete and hence unsatisfying. That is, in any particular realization, one or the other horse must emerge victorious, but one wants to know whether the victory is statistically significant. That is, one wants to know whether a victory “in sample” was merely good luck, or truly indicative of a difference “in population.”

(emphasis mine)

Secondly how would one go about interpreting the following Diebold Mariano test I've conducted. Would the below be correct?

Strictly speaking, you never accept a hypothesis (be it null or alternative), you can only fail to reject.
Also note that Forecast 1 is less accurate than forecast 2 is the same as Forecast 2 is more accurate than Forecast 1 (of course you did not mean that).
Other than that, I do not know what the particular software means by >prog (with a spelling error) or <prob, but it indeed looks as if they refer to outcomes of two one-sided $t$-tests. You should be able to figure out which forecast is better by looking at the loss values, then you will know clearly what forecasts these test result correspond to.

References:

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.