Mean absolute scaled error (MASE) is a measure of forecast accuracy proposed by Koehler & Hyndman (2006).
$$MASE=\frac{MAE}{MAE_{in-sample, \, naive}}$$
where $MAE$ is the mean absolute error produced by the actual forecast;
while $MAE_{in-sample, \, naive}$ is the mean absolute error produced by a naive forecast (e.g. no-change forecast for an integrated $I(1)$ time series), calculated on the in-sample data.
(Check out the Koehler & Hyndman (2006) paper for a precise definition and formula.)
$MASE>1$ implies that the actual forecast does worse out of sample than a naive forecast did in sample, in terms of mean absolute error. Thus if mean absolute error is the relevant measure of forecast accuracy (which depends on the problem at hand), $MASE>1$ suggests that the actual forecast should be discarded in favour of a naive forecast if we expect the out-of-sample data to be quite like the in-sample data (because we only know how well a naive forecast performed in sample, not out of sample).
Question:
$MASE=1.38$ was used as a benchmark in a forecasting competition proposed in this Hyndsight blog post. Shouldn't an obvious benchmark have been $MASE=1$?
Of course, this question is not specific to the particular forecasting competition. I would like some help on understanding this in a more general context.
My guess:
The only sensible explanation I see is that a naive forecast was expected to do quite worse out of sample than it did in sample, e.g. due to a structural change. Then $MASE<1$ might have been too challenging to achieve.
References:
- Hyndman, Rob J., and Anne B. Koehler. "Another look at measures of forecast accuracy."Another look at measures of forecast accuracy." International journal of forecasting 22.4 (2006): 679-688.
- Hyndsight blog post.
