Background

Denote the forecasted value by a Time Series ML Model as \begin{align} \hat{y}_{t+\tau} \, (t, \tau) \end{align} where $$t$$ is the current time-step, $$\tau$$ is the time-step ahead. The subscript $$t + \tau$$ is the corresponding forecast time-step.

For time series forecast ML Models, we have a hyper parameter of prediction_length (e.g. DeepAR or Temporal Fusion Transformer) which is the maximum of $$\tau$$

Suppose we set prediction_length = 2 as the model hyper-parameter. That means $$\tau \in \{1, 2\}$$

At time-step $$t = 0$$, the model prediction at $$t = 2$$ is denoted as $$\hat{y}_{2}(0, 2)$$.

As time pass, now we are at time-step $$t = 1$$, we collected the ground-truth $$y(t=1)$$. Then the prediction at $$t = 2$$ would be updated, and it becomes $$\hat{y}_{2}(1, 1)$$

Question

As time pass again, now we are at time-step $$t = 2$$, we now collected the ground-truth $$y_{2}$$.

My question is, how should I evaluate the model performance in predicting the value at $$t = 2$$? Should the residual be $$y_{2} - \hat{y}_{2}(0,2)$$ or $$y_{2} - \hat{y}_{2}(1, 1)$$?

I am not sure in general, what error metrics should I use to evaluate the time series model involving prediction_length hyper-parameters

It depends on whether you are mainly interested in 1-step-ahead forecasts (use $$y_{2} - \hat{y}_{2}(1, 1)$$) or in 2-step-ahead forecasts (use $$y_{2} - \hat{y}_{2}(0,2)$$). You might even be interested in total forecasts and total actuals over a two-step horizon (then use $$y_1+y_2-(\hat{y}_{1}(0, 1)+\hat{y}_{2}(0, 2)$$).
• For instance, you might be forecasting for inventory control or service provision. If you need to set up a work schedule for a call center two days in advance, you are mainly interested in the 2-step-ahead forecast (use $$y_{2} - \hat{y}_{2}(0,2)$$).
• Or you might be forecasting for supermarket replenishment, writing orders today after the store closes, with product arriving tomorrow morning before the store opens, and the next order arriving two days later. In this case, you are mainly interested in total demand and total forecasts (use $$y_1+y_2-(\hat{y}_{1}(0, 1)+\hat{y}_{2}(0, 2)$$). (And you would really be more interested in a pinball loss for a quantile forecast. Actually, probably also in the call center example above.)