0
$\begingroup$

I have a time series regression that, given a set of lagged values, predicts all of the values from one step ahead to a given horizon. I'm trying to calculate the MASE for this, but am having a problem understanding how to apply it.

My understanding is that the MASE is the ratio of the MSE to a scaling factor, equal to the 'naive forecast'. The problem I have with this is that doing a 'naive' forecast for multiple-output seems difficult to quantify 'naively'.

For instance, if I have a set of values $1,2,..,n$, I may take in the first two $(1, 2)$ and want to output the series $3,..,n$. If this is the case then how can I use the "previous value" since this won't be available to the initial forecast? For instance, if I subtract the difference between:

$$\{4,5,..,n+1\} - \{3,4,..,n\}$$

Then sum them up, this "naive" forecast will have more information to forecast on than my learning algorithm would have, so it's not 'naive' at all. If, instead I simply take the previous value and project it out, then it seems like I may overstate the error of the naive forecast, especially for a long horizon.

I'm weighting the outputs of my forecast, so I've considered doing the same here, but it still seems like this could be result in overly generous error results.

Is there any better way to apply MASE to multi-output time series regression?

$\endgroup$
1
$\begingroup$

You might be overthinking things.

First of all, the MASE is defined as the mean absolute error, scaled by a benchmark, not the mean squared error. Nothing keeps you from scaling the MSE, but you shouldn't call the result the "MASE", it might confuse people - best to call a Mean Squared Scaled Error an MSSE.

Second, the MASE is just that: the MAE scaled by the MAE achieved by some benchmark method. In the original formulation of the MASE (Hyndman & Koehler, 2006, IJF, and see also here), the scaling factor was set to be the MAE of the naive no-change method, used in-sample. That is, you'd forecast each actual by the previous actual and calculate the MAE achieved by this method in-sample, and use this MAE as the denominator in the MASE.

However, what benchmark you use is not set in stone. Use whatever "simple" method makes sense. For instance, if you have seasonal data, using a naive seasonal method as a benchmark makes sense, e.g., forecasting next June's data by last June's observations.

In your case, you seem to want to forecast farther out than you have history, if I understand you correctly. That's not a problem. Just calculate the in-sample MAE of the naive no-change method, and use this as the benchmark. It doesn't matter whether this is based on fewer observations than your true holdout forecast. The idea of the MASE is simply to scale your error in a way that makes the MASE between different series at different scales comparable!

(Of course, if you truly want to forecast out 20 periods with only 5 historical periods' data, your forecasts might not be overly accurate. But I hope you won't be surprised by this, and anyway, it's not a feature of any particular accuracy measure.)

$\endgroup$
  • $\begingroup$ Ah, that makes a lot of sense! Thank you! Apologies for mixing up MAE and MSE, I'm utilizing both and mixed them up here. I guess the question I have is then, how to compare the results of different series? For instance, if I get a MASE from my forecasting method and want to compare it against one of the GEFCOM methods, these are on different series at different scales. Is this not the utility? Is it taking different series at different scales on the same data set, for instance daily/weekly/monthly on the same hourly data? Or just to compare two algorithms doing the same regressions? $\endgroup$ – Nate Diamond May 1 '17 at 18:57
  • $\begingroup$ Well, the idea of the MASE is to tell how much a particular forecasting improves on the benchmark, in a relative sense. So you should certainly be able to compare the MASE of different series. This earlier question might be useful. $\endgroup$ – Stephan Kolassa May 1 '17 at 19:00
  • $\begingroup$ Ah, I think I get it. So in this case I could change my naive method to be "take the deltas from the same time in the previous day and apply them with the current value as the starting point" as a benchmark, assuming I've corrected the data for seasonality. So it seems to me that choosing a "better" benchmark would yield "worse" results, though that may be misleading because a small jump in results in one set could be very valuable compared to a large jump in another, which I think you highlighted in your answer there. Is this correct? I guess I'm worried about "cheating" in the loss measure. $\endgroup$ – Nate Diamond May 1 '17 at 19:07
  • $\begingroup$ You are exactly right! And no need to worry, as long as you take the same benchmark algorithm for all time series whose forecasts you are comparing based on MASE - you are still comparing apples to apples after all (the "apple" being each forecast's improvement over this benchmark method). $\endgroup$ – Stephan Kolassa May 2 '17 at 4:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.