Mean Absolute Scaled Error implementation on multistep time series forecast

Question

The formula for MASE can be found here: https://en.wikipedia.org/wiki/Mean_absolute_scaled_error

I am building a multi-step time series forecaster and I want to use MASE as a measure of prediction accuracy for a horizon of $n$ steps. These $n$ steps are part of the testing set, not the training set. I've trained the model and now I choose a starting point from my test set to begin the $n$-step forecast.

When calculating the naive mean absolute error (naive = using previous time step as next time step) should I calculate it on the same interval as the multistep forecast (which is comprised of test data), or on the entire training set?

Currently I calculate the naive mean absolute error on the entire training set, and I use that constant number for very multi-step MASE calculation. I still get MASE<1, which is encouraging, but I think perhaps this comparison doesn't make sense and I need to be comparing absolute forecast error and absolute naive error on the same interval every time. But this would mean that my accuracy would depend on each interval, which also doesn't seem ideal.

Answer 1

I think you should calculate the naive error on the same interval when comparing interval predictions and average the error over all the predictions thereafter, but this should be done using a naive forecast for each multi-step prediction that extends from $t=1$ to $t=n$ steps, i.e. take $Y(t)$ and use that as prediction for $Y(t+1), Y(t+2), ..., Y(t+n)$ (i.e. use a constant value as prediction). Consequently, your predictor will likely perform better than the naive forecast if $n > 1$ and the time series is not just a constant value.

If you use the naive forecast recursively to compare your multi-step forecast with, you will likely find that the naive forecast becomes better when $n$ increases. This is not a fair comparison however, because you will be comparing your predictor that has information about only $Y(t), Y(t-1), ...$ with a naive forecast that contains information up to $Y(t + n - 1)$.

Another option is to use aggregation to create a single step ahead forecast from your multi-step forecast, for example if you're forecasting daily sales and you predict 7 days ahead you aggregate the 7 forecast days to produce a weekly forecast. This aggregated weekly forecast can subsequently be compared to the naive forecast, in this case the aggregated actual sales of the previous week. This allows using the MASE in a normal way. I would not optimize to this aggregated metric directly though as the underlying forecasts may no longer make sense in that scenario.

To me at this point it makes more sense to use other measures such as RMSE to produce sensible multi-step ahead forecasts, maybe with the MASE based on aggregation used afterwards as an additional error metric to ensure that on a higher level the predictions are also relatively accurate.

Finally, I don't understand this sentence:

These $n$ steps are part of the testing set, not the training set.

The number of prediction steps are not a function of your data set? It is just a choice you make.

Disclaimer: I am certainly no expert on this topic and I am only just learning these concepts myself.