4
$\begingroup$

When applying the Diebold-Mariano test to test for predictive accuracy we need to specify a loss differential. For instance the loss differential $d$ in terms of the mean absolute error (MAE) is

$d = abs(e_1) - abs(e_2)$

where $e_1$, $e_2$ are the errors of two competing forecasts and are nobs-by-1 matrices.

When we want to compare the forecasts in terms of root mean square error (RMSE), how the loss differential should be defined? My intuition says

$d = (e_1^2)^{0.5} - (e_2^2)^{0.5},$

but this is exactly as the MAE above. Do we maybe need to use the MSE in this case?

What's missing?

$\endgroup$
1
$\begingroup$

I think root mean squared error (RMSE) is not a valid loss function when defined on a single forecast error, as there it reduces to the absolute error (as you note yourself).

The Diebold-Mariano test considers two random variables $e_1$ and $e_2$ that generate forecast errors $e_{1,i}$ and $e_{2,i}$ for $i=1,\dots,n$ where $n$ is the number of times a forecast is made (and an error committed). The test aims at assessing whether the expectations of some function (absolute value, square, ...) of $e_1$ and $e_2$ are equal.

Meanwhile, the "mean" in "root mean squared error" refers to averaging over a number of elements, thus it takes more than one error as its argument. As such it does not fit into the framework of the Diebold-Mariano test.

But you could also think as follows. Consider a group of errors (e.g. $5$ consecutive errors) as a single element; thus instead of two scalar random variables $e_1$ and $e_2$ you have two vector-valued random variables $\varepsilon_1$ and $\varepsilon_2$. Take RMSE as your loss function. If you have a sample of $n$ realizations of $\varepsilon_1$ and $\varepsilon_2$ (corresponding to $5n$ "individual errors"), then you can use the Diebold-Mariano test to assess whether the population RMSEs of $\varepsilon_1$ and $\varepsilon_2$ are the same.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.