2
$\begingroup$

This is my first time doing time series forecasting, so I am sorry for any inconsistencies in my question. But I have two different models that I want to compare. On Wikipedia, I read about Mean Absolute Scaled Error and that it can be used for the Diebold-Mariano test for a one-step forecast to test the statistical significance of the difference between two sets of forecasts.

I have calculated MASE for two different forecasts and I want to compare them, but I can not find information on how to compute the test statistic.

$\endgroup$

1 Answer 1

2
$\begingroup$

The Diebold-Mariano (DM) test is about the expected value of loss from one forecast vs. another forecast, $\mathbb{E}[L(e_1)]$ vs. $\mathbb{E}[L(e_2)]$. The $H_0$ is $\mathbb{E}[L(e_1)]=\mathbb{E}[L(e_2)]$ or equivalently, $\mathbb{E}[L(e_1)-L(e_2)]=0$. In words, the expected value of loss differential is zero.

The DM test applies on raw losses such as absolute error or squared error. The test statistic is the usual $t$ statistic applied on the loss differential $L(e_{1,t})-L(e_{2,t})$ with $t=1,\dots,n$: $$ DM=\frac{\bar{x}}{s_x/\sqrt{n}} $$ where $\bar{x}$ is the mean loss differential, $s_x$ is the estimated standard deviation of the loss differential and $n$ is the sample size. The DM test does not apply on averages such as mean absolute error (MAE) or mean squared error (MSE). Thus, it does not apply on MASE, as MASE is just a scaled version of MAE. If you just have two MASE values (one for each forecast), you cannot run the DM test; you do not have the information needed for obtaining the test statistic, e.g. you do not have $s_x$.

However, MAE is the empirical estimate of the expected loss corresponding to absolute errors: $\text{MAE}=\hat{\mathbb{E}}[|e|]$. So, you can simply run the DM test on absolute errors $L(e_i)=|e_i|$ for $i=1,2$. Your $H_0$ will be $\mathbb{E}[|e_1|]=\mathbb{E}[|e_2|]$, i.e. that the expected absolute errors from the two forecasts are equal in population.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.