Interpretation of scaled error measures

Question

can someone give me an explanation on how one would interpret the result of a scaled error measure. For example the Mean Absolute Scaled Measure (MASE). The numerator is the mean absolute error and the denominator the mean absolute error of a benchmarking method.

While I know that a MASE < 1 means that the forecasting method is better than the naive one, and MASE > 1 that it is worse, I still have a difficult time trying to interpret the results.

Does the result say nothing about how the forecast performed in regards to the actual result? Does it only compare the result to a benchmarking method?

Let's say you are working for a company that makes sales predictions, now you want to present your performance to a customer. What would a e.g. 0,8 MASE say about your overall performance? Does it just say you're better than the naive method because it is < 1?

The MAPE for example would say that you are off by 80% if the MAPE were 0,8, but then again the mape is heavily unreliable, this is why I opted for scaled measures. I just find it so difficult to interpret the result in a 'customer friendly' way.

Thanks for your help!

Answer 1

Your interpretation is exactly correct: a MASE <1 simply says that our method performs better than the benchmark method, in terms of the MAE. (One could also look at scaled squared errors, in which case we would compare MSEs between the focal and the benchmark method.) As such, the MASE does not tell you anything about "absolute" forecast accuracy, just about whether we improved on the benchmark.

In addition, note that how exactly we calculate the benchmark MAE matters. For instance, in the original MASE formulation by Hyndman & Koehler (2006), the benchmark error, i.e., the denominator, was the MAE of the random walk forecast in-sample. So for the MASE, we compare the out-of-sample performance of our focal forecasting method to the benchmark in-sample. See Interpretation of mean absolute scaled error (MASE) on what kind of confusion may result.

You are right to be worried about interpretation. Clients often don't care that much about whether we have improved on a benchmark. (Even if such an improvement over a simple benchmark is by no means a foregone conclusion.) Then again, "absolute" accuracy measures may not be all that informative, either.

I personally like "forecast implication metrics": what are the implications of different forecasts? Forecast A may be more accurate than forecast B, but if subsequent processes mean that they have the same implications (e.g., because we are using forecasts as an input to a production planning process, but logistical constraints mean that both forecasts will lead to the same production plan), then in terms of actual outcomes, both forecasts are equally good.

Better forecasts won't earn you money. Better production plans, capacity utilization, or stock control will. Forecast accuracy is only a means to an end. And yes, this implies that interpreting forecast accuracy without looking at the larger picture is short-sighted.