mean(abs(pred - true)) / mean(abs(true))
I defined above metric that works well for measuring reconstruction error for zero-mean signal data (1D); does it have a name? It's not MAPE.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
It's very closely related to MAD/Mean, where you just take the mean of the series for the denominator, without taking absolute values. If your series is nonnegative, the two notions coincide.
MAD/Mean is a weighted MAPE, with the actuals as the weighting (Kolassa & Schütz, 2007, Foresight). Your MAD/MAV analogously is a weighted MAPE with the absolute values of the actuals as the weighting.
Whether or not this KPI "works well"... well, if your series is stationary, then it is equivalent to the plain MAD, which is minimized by the median. So minimizing this KPI will incentivize you to report the predictive median. If your series is symmetric, then this will be the same as the mean. If not, your predictions will be biased. That's fine if you want it, but it should be kept in mind (Kolassa, 2020, IJF). The discussion at What are the shortcomings of the Mean Absolute Percentage Error (MAPE)? is related.
answered Nov 4, 2020 at 6:57
-
$\begingroup$ Thanks for the info. The sole motivation is scale-invariance; reconstructing a signal with twice the amplitude gives twice the error, even though "relative" error is the same, and I can't just take mean of non-absolute signal as that's ~0. I also asked here; if there's really no name for it, perhaps time to coin one. $\endgroup$ Commented Nov 4, 2020 at 16:16
-
$\begingroup$ Hmm, important problem in this metric: not robust to sparsity, or simply signal duration. E.g. if we simply add zeros to an existing signal, and assume the prediction signal maps them perfectly, then MAE/MAV will begin to increase. I've settled for the RMS (i.e. L2 norm) instead. $\endgroup$ Commented Nov 5, 2020 at 0:48
-
$\begingroup$ Yes, that is precisely the fact that MAE is minimized by the median. If you add probability mass at zero, then the median will tend towards zero. In extreme cases, when half or more of your mass is zero, the flat zero prediction is MAE-optimal. This used to confuse people who do intermittent demand forecasting (e.g., sales of rarely sold items at supermarkets). $\endgroup$ Commented Nov 5, 2020 at 7:18
-
$\begingroup$ Indeed, though I was in fact referring to the normalization (i.e. denominator), and it turns out I haven't really solved the problem. $\endgroup$ Commented Nov 5, 2020 at 18:50