Mean absolute percentage error with respect to predictions

Question

A friend of mine has suggested that instead of using mean absolute percentage error, i.e.

$$ \frac{1}{N}\sum_{i=0}^N \left| \frac{A_i - P_i}{A_i} \right| $$

(where $A_i$ denotes an actual value, $P_i$ denotes a predicted value, and $N$ the number of observations), to modify it so that the predicted amount is in the denominator, i.e.

$$ \frac{1}{N}\sum_{i=0}^N \left| \frac{A_i - P_i}{P_i} \right| $$

They've said that like this we penalise underforecasting more than overforecasting.

This seems strange to me, though I'm having trouble articulating why. What problems might there be with this latter error metric (as compared with regular MAPE on top)?

Answer 1

First off, using the prediction in the denominator is a reasonably common idea, see the survey by Green & Tashman (2009). Also, some people use the average of the prediction and the actual, which is commonly called a "symmetric" MAPE (see What is “symmetry” in evaluation metrics).

That said, "we penalize underforecasting more than overforecasting" is an accurate description. Note, though, that an alternative description is that "the optimal forecast under this MAPE variant is higher than the expectation". That is, using this MAPE will reward you for reporting systematically biased forecasts. Of course, this is completely analogous to the situation with the "normal" MAPE, which will reward you for biasing your forecasts low: What are the shortcomings of the Mean Absolute Percentage Error (MAPE)? Our thread on Minimizing symmetric mean absolute percentage error (SMAPE) addresses the bias that the sMAPE induces.

How strong the bias turns out will of course depend on the (unknown) future predictive distribution. I like to simulate stuff like this: randomly generate many iid future outcomes, and find out which point forecast will minimize your MAPE variant in expectation.

I personally would say that a major drawback of all kinds of MAPEs is that we don't know which functional of the future distribution will minimize the error measure. That is, we don't know what we are shooting for. I would say that if we want an unbiased expectation forecast, we should use (a variant of) the MSE, and if we want a given quantile forecast, we should use the appropriate quantile loss function. In other words, we should first figure out which functional of the unknown future density we want to elicit, then tailor or error measure accordingly.