"Percentage" alternatives to MAPE

Question

I'm aware of the problems of MAPE as a measurement, and particularly it's uselessness in the event of a time series where 0 is one of the many values of y.

The downside to ditching MAPE in favour of something like RMSE is that that measurement is less comparable where the 'scale' of the y values involved changes. For example with MAPE I can look at model A, where sum(y) = 1000 and model B where sum(y) = 1000000 and say they're comparable when MAPE = 0.1 for both. With RMSE, an error of 100 for A vs 100000 for B makes it less clear.

What alternative evaluation metrics retain that comparability, but don't have the same problems as MAPE (particularly the inability to handle y=0)? My instinct is something like RMSE / mean of y, but I'm not certain whether that's particularly valid.

EDIT: Turns out yes, it is. It's called normalised root mean square error. I'd love to hear of any others though

Answer 1

You may look into scaled versions of the MAE and RMSE, namely the MASE and RMSSE proposed by Hyndman and Koehler (2006). Besides being scale-invariant they behave well when $y=0$ and are symmetric (that is, penalizing positive and negative errors equally). Furthermore, these metrics are asymptotically normal (see Franses (2016)) so that they can be used within a testing setup if desired.

Answer 2

Although very intuitive and was preferred commonly (me, too), the only problem you may face with MAPE is not just the case where $y=0$. Especically when the predicted series is positive, e.g. CLTV, when $\hat{y}<y$, (an underestimate), the percentage error cannot be greater than $1$. In overestimates, $\hat{y}>2y$ guarantees a MAPE with larger than $1$. So, it implicitly favours for underestimates. To deal with such issues, sometimes we can resort to SMAPE (which was proposed by Makridakis, or by Armstrong with the name adjusted MAPE), in which we take the average (sometimes sum) of $y$ and $\hat{y}$ for the denominator.