3
$\begingroup$

After looking at Hyndman and Koehler, 2006 and applying the metric to my own data, I have been convinced that MASE is a better metric for evaluating forecast error than the method I had been previously using (MAPE), at least for short horizon forecasts. However, I am struggling with how to interpret the value of the MASE at longer horizons.

At short horizons it makes sense, it is effectively quantifying how much better (or worse) your forecast did than one of the simplest forecasting methods (the naive/seasonal naive). If your MASE is greater than 1 then, you may want to be suspect that your forecasting method is adding value (assuming there have been no major changes in your data between the in sample and out of sample values).

Where I get confused is for longer horizon forecasts. Imagine I have a situation like below where I have a random walk series I want to forecast (black). For convenience, I will be looking at in sample errors. In cyan, I have plotted the rolling horizon 20 naive forecast for the series as well at the one step (horizon 1) naive forecast in orange which would be used to compute the denominator of the MASE.

random walk example

The horizon 20 forecast is much worse for obvious reasons, it has less information than the horizon 1 forecast. Accordingly, the MASE reflects this and has a value of 2.88. However, we know that the naive forecast is the best forecast we can make for a random walk series, so it is the forecast we should be using even though the MASE is "bad".

It is easy to know in this circumstance that the MASE being greater than 1 is ok because we have a theoretical grounding, but this will not be the case for most series we want to evaluate. How do we properly interpret the MASE for these longer horizon applications then? Should we create a hMASE which would be a horizonal dependent MASE to match the horizon of interest changing it from

$$ MASE = \frac{\sum_{t=1}^{T} \left| e_t \right|}{\frac{T}{T-1}\sum_{t=2}^T \left| Y_t-Y_{t-1}\right|} $$

to

$$ hMASE = \frac{\sum_{t=1}^{T} \left| e_t \right|}{\frac{T}{T-h}\sum_{t=h+1}^T \left| Y_t-Y_{t-h}\right|} $$

or would we need to also calculate the MASE for the naive forecast and use that as a comparison? Is there some other method I am not thinking of? What If I am evaluating both short and long term forecasts? Would I include multiple errors for each in sample data point (one for each horizon) in the denominator of my MASE?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The MASE compares "your" forecast against a naive benchmark forecast calculated in-sample.

The original paper by Hyndman & Kohler (2006) chose the one-step ahead random walk forecast as this benchmark, but this is not set in stone. For instance, if you have an obviously seasonal time series, then the random walk makes little sense as a benchmark, and it would be better to use a seasonal naive forecast (i.e., the value from one season back), like Hyndman does in this paper mentioned in this earlier CV question about the MASE. Rob Hyndman doesn't write so explicitly in his online textbook, but he has repeatedly made this point in discussions about the MASE.

In your case, I'd use a naive 20-step-ahead forecast as the benchmark, i.e., your $hMASE$. Just make it clear in your writeup what the benchmark is that you are comparing "your" forecast against.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for this answer. I was aware that the seasonal naive forecast could be used rather than naive one, but I was unaware that different horizons (or even forecasting methods) were also acceptable. In this case, how is MASE different than MRAE? Is it because in MASE the evaluated errors (e) are divided by the mean of the reference errors (e') ie mean(e / mean(e')) instead of evaluated pairwise ie mean(e / e')? $\endgroup$
    – Barker
    Commented Jan 17, 2017 at 21:28
  • 1
    $\begingroup$ That is one difference. Another one is that the MASE denominator is calculated in-sample, not in the holdout sample. This is frequently confused. $\endgroup$
    – Stephan Kolassa
    Commented Jan 17, 2017 at 21:35
  • $\begingroup$ That makes sense. So just to make sure I am thinking about this right, I shouldn't be thinking of MASE as a comparison to a set "gold standard" but rather a metric for comparing errors to a user defined baseline. I can set up the experiment for generating the baseline forecasts however I need to best compare to the out of sample forecasts, so long as the errors they generate are averaged over the in sample data. $\endgroup$
    – Barker
    Commented Jan 17, 2017 at 21:51
  • 1
    $\begingroup$ Exactly. If you do have a gold standard, then by all means do use that (and extremely simple methods, like a random walk, often make for a very good gold standard), but use whatever makes sense as a baseline. $\endgroup$
    – Stephan Kolassa
    Commented Jan 17, 2017 at 22:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.