21
$\begingroup$

Mean absolute scaled error (MASE) is a measure of forecast accuracy proposed by Koehler & Hyndman (2006).

$$MASE=\frac{MAE}{MAE_{in-sample, \, naive}}$$

where $MAE$ is the mean absolute error produced by the actual forecast;
while $MAE_{in-sample, \, naive}$ is the mean absolute error produced by a naive forecast (e.g. no-change forecast for an integrated $I(1)$ time series), calculated on the in-sample data.

(Check out the Koehler & Hyndman (2006) paper for a precise definition and formula.)

$MASE>1$ implies that the actual forecast does worse out of sample than a naive forecast did in sample, in terms of mean absolute error. Thus if mean absolute error is the relevant measure of forecast accuracy (which depends on the problem at hand), $MASE>1$ suggests that the actual forecast should be discarded in favour of a naive forecast if we expect the out-of-sample data to be quite like the in-sample data (because we only know how well a naive forecast performed in sample, not out of sample).

Question:

$MASE=1.38$ was used as a benchmark in a forecasting competition proposed in this Hyndsight blog post. Shouldn't an obvious benchmark have been $MASE=1$?

Of course, this question is not specific to the particular forecasting competition. I would like some help on understanding this in a more general context.

My guess:

The only sensible explanation I see is that a naive forecast was expected to do quite worse out of sample than it did in sample, e.g. due to a structural change. Then $MASE<1$ might have been too challenging to achieve.

References:

$\endgroup$
  • $\begingroup$ In his blog post, Rob notes where this benchmark comes from: "These thresh­olds are the best per­form­ing meth­ods in the analy­sis of these data described in Athana­sopou­los et al (2010)." Have you looked at the Athanosopoulos paper? $\endgroup$ – Stephan Kolassa Nov 17 '14 at 11:28
  • $\begingroup$ I am a bit puzzled by "your guess": a structural change would mean that the sophisticated forecast would be based on partly irrelevant past data, indeed. But how a structural break would affect a "no-change" forecast depends on the break. If for example we are looking at a random walk with drift, and the structural break means that the drift, the constant term, just got lower, then the "no-change" forecast will perform better after the break, than before it. $\endgroup$ – Alecos Papadopoulos Nov 17 '14 at 12:22
  • $\begingroup$ @AlecosPapadopoulos: you are right. However, I meant the out-of-sample data being quite different from the in-sample-data as a necessary, but not a sufficient condition for expecting $MASE>>1$. Perhaps I did not express myself correctly. $\endgroup$ – Richard Hardy Nov 17 '14 at 12:54
  • $\begingroup$ @StephanKolassa: I skimmed through the paper and did not find a good explanation. Perhaps I should read it more carefully. But my questions is intended to be more general than that. I am not particularly interested in that instance, I just presented it as an example. I am seeking intuition about $MASE$. $\endgroup$ – Richard Hardy Nov 17 '14 at 12:57
  • $\begingroup$ Related: stats.stackexchange.com/questions/124955/… $\endgroup$ – Stephan Kolassa Nov 21 '14 at 20:44
15
$\begingroup$

In the linked blog post, Rob Hyndman calls for entries to a tourism forecasting competition. Essentially, the blog post serves to draw attention to the relevant IJF article, an ungated version of which is linked to in the blog post.

The benchmarks you refer to - 1.38 for monthly, 1.43 for quarterly and 2.28 for yearly data - were apparently arrived at as follows. The authors (all of them are expert forecasters and very active in the IIF - no snake oil salesmen here) are quite capable of applying standard forecasting algorithms or forecasting software, and they are probably not interested in simple ARIMA submission. So they went and applied some standard methods to their data. For the winning submission to be invited for a paper in the IJF, they ask that it improve on the best of these standard methods, as measured by the MASE.

So your question essentially boils down to:

Given that a MASE of 1 corresponds to a forecast that is out-of-sample as good (by MAD) as the naive random walk forecast in-sample, why can't standard forecasting methods like ARIMA improve on 1.38 for monthly data?

Here, the 1.38 MASE comes from Table 4 in the ungated version. It is the average ASE over 1-24 month ahead forecasts from ARIMA. The other standard methods, like ForecastPro, ETS etc. perform even worse.

And here, the answer gets hard. It is always very problematic to judge forecast accuracy without considering the data. One possibility I could think of in this particular case could be accelerating trends. Suppose that you try to forecast $\exp(t)$ with standard methods. None of these will capture the accelerating trend (and this is usually a Good Thing - if your forecasting algorithm often models an accelerating trend, you will likely far overshoot your mark), and they will yield a MASE that is above 1. Other explanations could, as you say, be different structural breaks, e.g., level shifts or external influences like SARS or 9/11, which would not be captured by the non-causal benchmark models, but which could be modeled by dedicated tourism forecasting methods (although using future causals in a holdout sample is a kind of cheating).

So I'd say that you likely can't say a lot about this withough looking at the data themselves. They are available on Kaggle. Your best bet is likely to take these 518 series, hold out the last 24 months, fit ARIMA series, calculate MASEs, dig out the ten or twenty MASE-worst forecast series, get a big pot of coffee, look at these series and try to figure out what it is that makes ARIMA models so bad at forecasting them.

EDIT: another point that appears obvious after the fact but took me five days to see - remember that the denominator of the MASE is the one-step ahead in-sample random walk forecast, whereas the numerator is the average of the 1-24-step ahead forecasts. It's not too surprising that forecasts deteriorate with increasing horizons, so this may be another reason for a MASE of 1.38. Note that the Seasonal Naive forecast was also included in the benchmark and had an even higher MASE.

$\endgroup$
  • $\begingroup$ Great answer! Thanks for the concise summary of the original paper (it will serve as a helpful shortcut to all of the uninitiated). It seems that the main idea behind your answer does not conflict with my guess (but rather extends it); there is something special out of sample that the in-sample naive forecast error underestimates. $\endgroup$ – Richard Hardy Nov 17 '14 at 13:38
2
$\begingroup$

Not an answer, but a plot following Stephan Kolassa's call to "look at these series".
Kaggle tourism1 has 518 yearly time series, for which we want to predict the last 4 values:

enter image description here

The plot shows the errors from the "naive" constant predictor, here $5^{th}$ last:
$\qquad Error4( y ) \equiv {1 \over 4} \sum_ {last\ 4} |y_i - y_{-5}| $
The numbers in the corners, 81 12 ..., are $Error4(y)$ as % of range, and $length(y)$.
The 3 rows are the 10 worst, 10 in the middle, and 10 best of all 518 yearly time series.

Obviously, very short series -- 12 11 7 7 7 ... in the top row -- are hard to predict: no surprise.
(Athanasopoulos, Hyndman, Song and Wu, The Tourism Forecasting Competition (2011, 23p) used 112 of the 518 yearly series, but I don't see which ones.)

Are there other, newer collections of time series since 2010, that might be worth looking at ?

$\endgroup$
  • $\begingroup$ Thanks! I do not know the answer to your last question. $\endgroup$ – Richard Hardy Jun 9 '15 at 17:14
  • 1
    $\begingroup$ @denis: just saw your question - you may want to ask for data at OpenData.SE. $\endgroup$ – Stephan Kolassa Feb 12 '16 at 10:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.