What error metrics to use for time series model with time-step ahead hyper-parameter?

Question

Background

Denote the forecasted value by a Time Series ML Model as \begin{align} \hat{y}_{t+\tau} \, (t, \tau) \end{align} where $t$ is the current time-step, $\tau$ is the time-step ahead. The subscript $t + \tau$ is the corresponding forecast time-step.

For time series forecast ML Models, we have a hyper parameter of prediction_length (e.g. DeepAR or Temporal Fusion Transformer) which is the maximum of $\tau$

Suppose we set prediction_length = 2 as the model hyper-parameter. That means $\tau \in \{1, 2\}$

At time-step $t = 0$, the model prediction at $t = 2$ is denoted as $\hat{y}_{2}(0, 2)$.

As time pass, now we are at time-step $t = 1$, we collected the ground-truth $y(t=1)$. Then the prediction at $t = 2$ would be updated, and it becomes $\hat{y}_{2}(1, 1)$

Question

As time pass again, now we are at time-step $t = 2$, we now collected the ground-truth $y_{2}$.

My question is, how should I evaluate the model performance in predicting the value at $t = 2$? Should the residual be $y_{2} - \hat{y}_{2}(0,2)$ or $y_{2} - \hat{y}_{2}(1, 1)$?

I am not sure in general, what error metrics should I use to evaluate the time series model involving prediction_length hyper-parameters

Answer 1

It depends on whether you are mainly interested in 1-step-ahead forecasts (use $y_{2} - \hat{y}_{2}(1, 1)$) or in 2-step-ahead forecasts (use $y_{2} - \hat{y}_{2}(0,2)$). You might even be interested in total forecasts and total actuals over a two-step horizon (then use $y_1+y_2-(\hat{y}_{1}(0, 1)+\hat{y}_{2}(0, 2)$).

And this in turn should be driven by what you plan on using the forecast for.

• For instance, you might be forecasting for inventory control or service provision. If you need to set up a work schedule for a call center two days in advance, you are mainly interested in the 2-step-ahead forecast (use $y_{2} - \hat{y}_{2}(0,2)$).
• Or you might be forecasting for supermarket replenishment, writing orders today after the store closes, with product arriving tomorrow morning before the store opens, and the next order arriving two days later. In this case, you are mainly interested in total demand and total forecasts (use $y_1+y_2-(\hat{y}_{1}(0, 1)+\hat{y}_{2}(0, 2)$). (And you would really be more interested in a pinball loss for a quantile forecast. Actually, probably also in the call center example above.)

Bottom line: the loss function should be governed by the problem the forecast should help you solve.