3
$\begingroup$

MAPE metric has problems when the actual value to be predicted is very small. In the extreme when the actual value is 0 then MAPE will be infinity (if the prediction is not exactly 0). What about this solution as illustrated with an example:

Suppose that the actual values are [10, 0, 20] and the corresponding predictions are [13, 2, 16].

Sum of Absolute Errors = 3 + 2 + 4 = 9
Sum of Actual Values = 10 + 0 + 20 = 30
Percentage error = (9/30)*100 = 30

What are the shortcomings of this metric? Also does this metric have a name (I could not find it)? Thanks

$\endgroup$
1

2 Answers 2

2
$\begingroup$

Your error is the sum of absolute deviations, divided by the sum of the actuals. If you divide both the numerator and the denominator by the number of samples, we see that it is equal to the mean absolute deviation (or error), divided by the mean of the actuals, or shorter MAD/Mean. We wrote an accessible paper on this (Kolassa & Schütz, Foresight, 2007) and how it can be interpreted as a weighted MAPE, where the weights are just the actuals.

Since you can't influence the actuals in the denominator, minimizing this MAD/Mean or wMAPE is equivalent to minimizing the MAE. Thus, this error measure will be minimized in expectation by the conditional median, not the conditional mean, so using this error measure may give you biased forecasts, especially if your data are conditionally asymmetrically distributed, e.g., lognormally or gamma.

$\endgroup$
2
$\begingroup$

There are many ways how you could normalize error metric, for example, you could divide it by a known constant that serves as a benchmark. What you are suggesting is similar to relative absolute error

$$ \mathrm{ RAE} = \frac{ \sum^N_{i=1} | \hat{\theta}_i - \theta_i | } { \sum^N_{i=1} | \overline{\theta} - \theta_i | } $$

but notice two differences:

  1. RAE normalized by the distance from the average, rather than sum. In such a case, you compare to a benchmark. With comparing to sum the error would be get smaller as sample size would grow and generally the values may be trickier to interpret than you assume.
  2. What if the true values sum to zero? It is totally possible, you just need to have the data with mean equal to zero for that. If they sum to small value you have the same problem as with MAPE.

Every "relative" error metric has some edge cases where it can give strange results or be hard to interpret, that is one of the reasons why people often prefer unnormalized metrics like means absolute error, or mean squared error, etc.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.