Multi-input, multi-output time series regression loss using MASE

Question

I have a time series regression that, given a set of lagged values, predicts all of the values from one step ahead to a given horizon. I'm trying to calculate the MASE for this, but am having a problem understanding how to apply it.

My understanding is that the MASE is the ratio of the MSE to a scaling factor, equal to the 'naive forecast'. The problem I have with this is that doing a 'naive' forecast for multiple-output seems difficult to quantify 'naively'.

For instance, if I have a set of values $1,2,..,n$, I may take in the first two $(1, 2)$ and want to output the series $3,..,n$. If this is the case then how can I use the "previous value" since this won't be available to the initial forecast? For instance, if I subtract the difference between:

$$\{4,5,..,n+1\} - \{3,4,..,n\}$$

Then sum them up, this "naive" forecast will have more information to forecast on than my learning algorithm would have, so it's not 'naive' at all. If, instead I simply take the previous value and project it out, then it seems like I may overstate the error of the naive forecast, especially for a long horizon.

I'm weighting the outputs of my forecast, so I've considered doing the same here, but it still seems like this could be result in overly generous error results.

Is there any better way to apply MASE to multi-output time series regression?

Answer 1

You might be overthinking things.

First of all, the MASE is defined as the mean absolute error, scaled by a benchmark, not the mean squared error. Nothing keeps you from scaling the MSE, but you shouldn't call the result the "MASE", it might confuse people - best to call a Mean Squared Scaled Error an MSSE.

Second, the MASE is just that: the MAE scaled by the MAE achieved by some benchmark method. In the original formulation of the MASE (Hyndman & Koehler, 2006, IJF, and see also here), the scaling factor was set to be the MAE of the naive no-change method, used in-sample. That is, you'd forecast each actual by the previous actual and calculate the MAE achieved by this method in-sample, and use this MAE as the denominator in the MASE.

However, what benchmark you use is not set in stone. Use whatever "simple" method makes sense. For instance, if you have seasonal data, using a naive seasonal method as a benchmark makes sense, e.g., forecasting next June's data by last June's observations.

In your case, you seem to want to forecast farther out than you have history, if I understand you correctly. That's not a problem. Just calculate the in-sample MAE of the naive no-change method, and use this as the benchmark. It doesn't matter whether this is based on fewer observations than your true holdout forecast. The idea of the MASE is simply to scale your error in a way that makes the MASE between different series at different scales comparable!

(Of course, if you truly want to forecast out 20 periods with only 5 historical periods' data, your forecasts might not be overly accurate. But I hope you won't be surprised by this, and anyway, it's not a feature of any particular accuracy measure.)