Interpretation of mean absolute scaled error (MASE)

Question

Mean absolute scaled error (MASE) is a measure of forecast accuracy proposed by Koehler & Hyndman (2006).

$$MASE=\frac{MAE}{MAE_{in-sample, \, naive}}$$

where $MAE$ is the mean absolute error produced by the actual forecast;
while $MAE_{in-sample, \, naive}$ is the mean absolute error produced by a naive forecast (e.g. no-change forecast for an integrated $I(1)$ time series), calculated on the in-sample data.

(Check out the Koehler & Hyndman (2006) paper for a precise definition and formula.)

$MASE>1$ implies that the actual forecast does worse out of sample than a naive forecast did in sample, in terms of mean absolute error. Thus if mean absolute error is the relevant measure of forecast accuracy (which depends on the problem at hand), $MASE>1$ suggests that the actual forecast should be discarded in favour of a naive forecast if we expect the out-of-sample data to be quite like the in-sample data (because we only know how well a naive forecast performed in sample, not out of sample).

Question:

$MASE=1.38$ was used as a benchmark in a forecasting competition proposed in this Hyndsight blog post. Shouldn't an obvious benchmark have been $MASE=1$?

Of course, this question is not specific to the particular forecasting competition. I would like some help on understanding this in a more general context.

My guess:

The only sensible explanation I see is that a naive forecast was expected to do quite worse out of sample than it did in sample, e.g. due to a structural change. Then $MASE<1$ might have been too challenging to achieve.

References:

• Hyndman, Rob J., and Anne B. Koehler. "Another look at measures of forecast accuracy." International journal of forecasting 22.4 (2006): 679-688.
• Hyndsight blog post.

Answer 1

In the linked blog post, Rob Hyndman calls for entries to a tourism forecasting competition. Essentially, the blog post serves to draw attention to the relevant IJF article, an ungated version of which is linked to in the blog post.

The benchmarks you refer to - 1.38 for monthly, 1.43 for quarterly and 2.28 for yearly data - were apparently arrived at as follows. The authors (all of them are expert forecasters and very active in the IIF - no snake oil salesmen here) are quite capable of applying standard forecasting algorithms or forecasting software, and they are probably not interested in simple ARIMA submission. So they went and applied some standard methods to their data. For the winning submission to be invited for a paper in the IJF, they ask that it improve on the best of these standard methods, as measured by the MASE.

So your question essentially boils down to:

Given that a MASE of 1 corresponds to a forecast that is out-of-sample as good (by MAD) as the naive random walk forecast in-sample, why can't standard forecasting methods like ARIMA improve on 1.38 for monthly data?

Here, the 1.38 MASE comes from Table 4 in the ungated version. It is the average ASE over 1-24 month ahead forecasts from ARIMA. The other standard methods, like ForecastPro, ETS etc. perform even worse.

And here, the answer gets hard. It is always very problematic to judge forecast accuracy without considering the data. One possibility I could think of in this particular case could be accelerating trends. Suppose that you try to forecast $\exp(t)$ with standard methods. None of these will capture the accelerating trend (and this is usually a Good Thing - if your forecasting algorithm often models an accelerating trend, you will likely far overshoot your mark), and they will yield a MASE that is above 1. Other explanations could, as you say, be different structural breaks, e.g., level shifts or external influences like SARS or 9/11, which would not be captured by the non-causal benchmark models, but which could be modeled by dedicated tourism forecasting methods (although using future causals in a holdout sample is a kind of cheating).

So I'd say that you likely can't say a lot about this withough looking at the data themselves. They are available on Kaggle. Your best bet is likely to take these 518 series, hold out the last 24 months, fit ARIMA series, calculate MASEs, dig out the ten or twenty MASE-worst forecast series, get a big pot of coffee, look at these series and try to figure out what it is that makes ARIMA models so bad at forecasting them.

EDIT: another point that appears obvious after the fact but took me five days to see - remember that the denominator of the MASE is the one-step ahead in-sample random walk forecast, whereas the numerator is the average of the 1-24-step ahead forecasts. It's not too surprising that forecasts deteriorate with increasing horizons, so this may be another reason for a MASE of 1.38. Note that the Seasonal Naive forecast was also included in the benchmark and had an even higher MASE.

Answer 2

Not an answer, but a plot following Stephan Kolassa's call to "look at these series".
Kaggle tourism1 has 518 yearly time series, for which we want to predict the last 4 values:

The plot shows the errors from the "naive" constant predictor, here $5^{th}$ last:
$\qquad Error4( y ) \equiv {1 \over 4} \sum_ {last\ 4} |y_i - y_{-5}| $
The numbers in the corners, 81 12 ..., are $Error4(y)$ as % of range, and $length(y)$.
The 3 rows are the 10 worst, 10 in the middle, and 10 best of all 518 yearly time series.

Obviously, very short series -- 12 11 7 7 7 ... in the top row -- are hard to predict: no surprise.
(Athanasopoulos, Hyndman, Song and Wu, The Tourism Forecasting Competition (2011, 23p) used 112 of the 518 yearly series, but I don't see which ones.)

Are there other, newer collections of time series since 2010, that might be worth looking at ?