0
$\begingroup$

I have two vector valued time series forecasts, and the components of the vector can be correlated. How do i significance test (ex. diebold mariano) for which is the better forecaster?

I found two ways in the literature:

  1. summarize the forecast at each time step with RMSE [however this doesn't take into account correlations between components]
  2. do multiple testing, one hypothesis for each component?
$\endgroup$
3
  • $\begingroup$ Could you please provide links to literature and also typeset with LaTeX the formulations. $\endgroup$ Commented Jul 10, 2021 at 3:45
  • $\begingroup$ What do you think about my answer? If it is helpful and clear, you may accept it by clicking on the tick mark to the left. Otherwise, you may ask for further clarification. This is how Cross Validated works. $\endgroup$ Commented Jul 13, 2021 at 9:56
  • $\begingroup$ I don't have enough reputation to cast a vote but I upvoted anyway. It says it records the response $\endgroup$
    – Dhamma K
    Commented Jul 13, 2021 at 22:11

1 Answer 1

0
$\begingroup$

The Diebold-Mariano test compares losses between two sets of forecasts (sets across units being forecast, not across dimensions). If you can define the loss of a vector-valued forecast (and you should be able to do that), you can apply the Diebold-Mariano test directly. If you cannot define the loss of a vector-valued forecast but can only define it for scalar-valued forecast, you cannot compare the forecasters directly. You can compare the forecasters on each of the tasks (one task per component of the vector), i.e. compare forecasts but not forecasters.

$\endgroup$
3
  • $\begingroup$ This answer is really helpful. I have some follow-up questions though. Why can you 'abstract away' the correlations between the different predictions with a loss function? Could the forecasters know about the loss function and tune their forecast (because of correlations to take advantage of some property of the loss? How does one go about choosing such a loss function for a particular task? Thanks $\endgroup$
    – Dhamma K
    Commented Jul 16, 2021 at 1:21
  • $\begingroup$ @DhammaK, regarding Why can you 'abstract away' the correlations?: Consider forecasting demand for items in a supermarket. Overforecasting will mean products being stored but unsold, underforecasting will mean empty shelves. Both cases are costly. In both cases the owner of the supermarket can calculate how costly it is relative to an ideal forecast. This difference is the loss, and it can be expressed in monetary terms. Obviously, there is correlation across products and their forecast errors, but this does not make it any harder to pin down the loss for any vector of forecast errors. $\endgroup$ Commented Jul 18, 2021 at 5:21
  • $\begingroup$ @DhammaK, regarding Could the forecasters know about the loss function and tune their forecast?: Yes, they could and they should. They cannot issue an optimal forecast if the loss function is unknown, because optimality depends on the loss function. But this is not problematic as far as I can tell. How does one go about choosing such a loss function for a particular task? Think about the owner of the supermarket. His/her task is rather straightforward. It is enough to know the cost of storage and the forgone profit from empty shelves. Neither is too difficult to figure out. $\endgroup$ Commented Jul 18, 2021 at 5:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.