1
$\begingroup$

Two models with differing assumptions each purport to provide a forecast for a vector of values 1 year in the future. Each model has been run at the start of each month for three years:

Run Date Model1 output Model2 output Actual Outcome
2021-01-01 $ \vec{y}_{M_11} $ $ \vec{y}_{M_21} $ $\vec{y}_{A1}$
2021-02-01 $ \vec{y}_{M_12} $ $\vec{y}_{M_22}$ $\vec{y}_{A2}$
... ... ... ...
2023-12-01 $ \vec{y}_{M_136}$ $\vec{y}_{M_236}$ (TBD)

Yielding (as of mid-February 2024) 26 outcome observations with high levels of autocorrelation, given that each forecast window will overlap with 11 others.

Is there a canonical Diebold-Mariano type test to compare the accuracy of the two models' forecasts given the issues engendered by the overlapping evaluations that won't involve sacrificing a large number of data points?

Improve this question
$\endgroup$
5
  • $\begingroup$ Hi: I don't quite understand the setup so I'm not sure if it applies but check out the Diebold-Mariano test. I forget the exact assumptions so make sure those align with your setup. $\endgroup$
    – mlofton
    Commented Feb 19 at 14:12
  • $\begingroup$ You can simply calculate MSEs or any other measure of forecast accuracy, getting 36 different MSEs for each model. Then you can compare them. The main point here is that because of the overlap, the MSEs will be strongly autocorrelated, which is why Diebold-Mariano-type hypothesis tests will likely not apply. Are you interested in hypothesis testing, or what exactly is your question? $\endgroup$
    – Stephan Kolassa
    Commented Feb 19 at 15:05
  • $\begingroup$ @StephanKolassa Yes, the autocorrelation is the main source of foreboding that prompted my question. I believe we would likely prefer mean absolute scaled error as a measure, but regardless: a hypothesis test that can account for the autocorrelation is what I think I'm after. (there will be 26 measures at present, as the last 10 points don't have observations yet, but that's neither here nor there - perhaps I shold have been clearer in the question. Thanks!) $\endgroup$
    – David
    Commented Feb 19 at 15:18
  • 1
    $\begingroup$ OK. I personally am not too keen on using the MASE (since it elicits the conditional median, not the conditional mean, which your models presumably optimize for, you may find Kolassa, 2020 helpful). But it looks like your core question is how to run a Diebold-Mariano type test with errors whose autocorrelation stems from an overlapping rolling evaluation. I don't see anything like this in the diebold-mariano-test tag. Maybe you want to edit your question to reflect this focus? $\endgroup$
    – Stephan Kolassa
    Commented Feb 19 at 15:32
  • $\begingroup$ @StephanKolassa thank you. Hopefully the edits clarify. I'll digest your paper and am not wedded to MASE, so have removed it from the question. $\endgroup$
    – David
    Commented Feb 19 at 15:52

1 Answer 1

Reset to default
1
$\begingroup$

For scalar-valued prediction losses, the Diebold-Mariano test is capable of handling multi-step ahead overlapping forecasts the errors (and losses) of which are autocorrelated by design. See Harvey et al. (1997). It employs HAC-robust standard errors instead of vanilla standard errors; the dm.test function in the forecast package has options for that (see h and varestimator). If scalar loss can be obtained from vector-valued forecasts, that is the solution.

References

  • Harvey, D., Leybourne, S., & Newbold, P. (1997). Testing the equality of prediction mean squared errors. International Journal of Forecasting, 13(2), 281-291.
Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the pointer and paper! For $n$-valued vectors, would it be logically consistent to treat them as $n$ separate forecasts, then? or would we need to really have 1 loss value per step - which also seems to be throwing away a bunch of information, just on a different axis. $\endgroup$
    – David
    Commented Feb 20 at 17:26
  • 1
    $\begingroup$ @David, I think this is mostly a subject-matter question, not a statistical one. If it makes more sense to conceptualize these as separate forecasts, you could do a Diebold-Mariano test for each of them. $\endgroup$
    – Richard Hardy
    Commented Feb 20 at 20:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.