103
$\begingroup$

The Mean Absolute Percentage Error () is a common accuracy or error measure for time series or other predictions,

$$ \text{MAPE} = \frac{100}{n}\sum_{t=1}^n\frac{|A_t-F_t|}{A_t}\%,$$

where $A_t$ are actuals and $F_t$ corresponding forecasts or predictions.

The MAPE is a percentage, so we can easily compare it between series, and people can easily understand and interpret percentages.

However, I hear that the MAPE has drawbacks. I'd like to understand these drawbacks better so I can make an informed decision about whether to use the MAPE or some alternative like the MSE (), the MAE () or the MASE ().

$\endgroup$
4
  • $\begingroup$ The denominator should be in absolute expression. Check this en.wikipedia.org/wiki/Mean_absolute_percentage_error $\endgroup$
    – pyaj
    Commented Dec 19, 2023 at 9:45
  • 1
    $\begingroup$ @pyaj: I would argue that if the absolute sign in the denominator makes a difference, i.e., if we do have negative values, then MAPE is even more dubious than otherwise, see this thread and this thread. $\endgroup$ Commented Dec 19, 2023 at 16:03
  • $\begingroup$ As shown in your links, I had positive and negative values in my dataset thus negative numbers so the absolute values at the denominator made more sens. $\endgroup$
    – pyaj
    Commented Dec 20, 2023 at 17:52
  • $\begingroup$ @StephanKolassa thanks for your input. I was wondering what your insight is about PI-related evaluation metrics and classic single-target related task or their potential shortcomings. please see my post you may have insightful input additionally and I can reconsider my answer. $\endgroup$
    – Mario
    Commented Mar 20 at 20:25

1 Answer 1

138
$\begingroup$

Shortcomings of the MAPE

  • The MAPE, as a percentage, only makes sense for values where divisions and ratios make sense. It doesn't make sense to calculate percentages of temperatures, for instance, so you shouldn't use the MAPE to calculate the accuracy of a temperature forecast.

  • If just a single actual is zero, $A_t=0$, then you divide by zero in calculating the MAPE, which is undefined.

    It turns out that some forecasting software nevertheless reports a MAPE for such series, simply by dropping periods with zero actuals (Hoover, 2006). Needless to say, this is not a good idea, as it implies that we don't care at all about what we forecasted if the actual was zero - but a forecast of $F_t=100$ and one of $F_t=1000$ may have very different implications. So check what your software does.

    If only a few zeros occur, you can use a weighted MAPE (Kolassa & Schütz, 2007), which nevertheless has problems of its own. This also applies to the symmetric MAPE (Goodwin & Lawton, 1999).

  • MAPEs greater than 100% can occur. If you prefer to work with accuracy, which some people define as 100%-MAPE, then this may lead to negative accuracy, which people may have a hard time understanding. (No, truncating accuracy at zero is not a good idea.)

  • Model fitting relies on minimizing errors, which is often done using numerical optimizers that use first or second derivatives. The MAPE is not everywhere differentiable, and its Hessian is zero wherever it is defined. This can throw optimizers off if we want to use the MAPE as an in-sample fit criterion.

    A possible mitigation may be to use the log cosh loss function, which is similar to the MAE but twice differentiable. Alternatively, Zheng (2011) offer a way to approximate the MAE (or any other quantile loss) to arbitrary precision using a smooth function. If we know bounds on the actuals (which we do when fitting strictly positive historical data), we can therefore smoothly approximate the MAPE to arbitrary precision.

  • The MAPE treats overforecasts differently than underforecasts. Suppose our forecast is $F_t=2$, then an actual of $A_t=1$ will contribute $\text{APE}_t=100\%$ to the MAPE, but an actual of $A_t=3$ will contribute $\text{APE}_t=33\%$. Minimizing the MAPE thus creates an incentive towards smaller $F_t$ - if our actuals have an equal chance of being $A_t=1$ or $A_t=3$, then we will minimize the expected MAPE by forecasting $F_t=1.5$, not $F_t=2$, which is the expectation of our actuals. The MAPE thus is lower for biased than for unbiased forecasts. Minimizing it may lead to forecasts that are biased low.

Especially the last bullet point merits a little more thought. For this, we need to take a step back.

To start with, note that we don't know the future outcome perfectly, nor will we ever. So the future outcome follows a probability distribution. Our so-called point forecast $F_t$ is our attempt to summarize what we know about the future distribution (i.e., the predictive distribution) at time $t$ using a single number. The MAPE then is a quality measure of a whole sequence of such single-number-summaries of future distributions at times $t=1, \dots, n$.

The problem here is that people rarely explicitly say what a good one-number-summary of a future distribution is.

When you talk to forecast consumers, they will usually want $F_t$ to be correct "on average". That is, they want $F_t$ to be the expectation or the mean of the future distribution, rather than, say, its median.

Here's the problem: minimizing the MAPE will typically not incentivize us to output this expectation, but a quite different one-number-summary (McKenzie, 2011, Kolassa, 2020). This happens for two different reasons.

  • Asymmetric future distributions. Suppose our true future distribution follows a stationary $(\mu=1,\sigma^2=1)$ lognormal distribution. The following picture shows a simulated time series, as well as the corresponding density.

lognormal

The horizontal lines give the optimal point forecasts, where "optimality" is defined as minimizing the expected error for various error measures.

We see that the asymmetry of the future distribution, together with the fact that the MAPE differentially penalizes over- and underforecasts, implies that minimizing the MAPE will lead to heavily biased forecasts. (Here is the calculation of optimal point forecasts in the gamma case.)

  • Symmetric distribution with a high coefficient of variation. Suppose that $A_t$ comes from rolling a standard six-sided die at each time point $t$. The picture below again shows a simulated sample path:

die roll

In this case:

  • The dashed line at $F_t=3.5$ minimizes the expected MSE. It is the expectation of the time series.

  • Any forecast $3\leq F_t\leq 4$ (not shown in the graph) will minimize the expected MAE. All values in this interval are medians of the time series.

  • The dash-dotted line at $F_t=2$ minimizes the expected MAPE.

We again see how minimizing the MAPE can lead to a biased forecast, because of the differential penalty it applies to over- and underforecasts. In this case, the problem does not come from an asymmetric distribution, but from the high coefficient of variation of our data-generating process.

This is actually a simple illustration you can use to teach people about the shortcomings of the MAPE - just hand your attendees a few dice and have them roll. See Kolassa & Martin (2011) for more information.

  • Finally, you should not use the MAPE, because by definition, the MAPE is a Mean APE... and nobody likes Mean APEs.

    A picture of angry chimpanzees in business suits arguing

    Image credit: Ivan Svetunkov. Used with kind permission. Please attribute to him if you reuse it.

Related CrossValidated and Data Science questions

R code

Lognormal example:

mm <- 1
ss.sq <- 1
SAPMediumGray <- "#999999"; SAPGold <- "#F0AB00"

set.seed(2013)
actuals <- rlnorm(100,meanlog=mm,sdlog=sqrt(ss.sq))

opar <- par(mar=c(3,2,0,0)+.1)
    plot(actuals,type="o",pch=21,cex=0.8,bg="black",xlab="",ylab="",xlim=c(0,150))
    abline(v=101,col=SAPMediumGray)

    xx <- seq(0,max(actuals),by=.1)
    polygon(c(101+150*dlnorm(xx,meanlog=mm,sdlog=sqrt(ss.sq)),
      rep(101,length(xx))),c(xx,rev(xx)),col="lightgray",border=NA)

    (min.Ese <- exp(mm+ss.sq/2))
    lines(c(101,150),rep(min.Ese,2),col=SAPGold,lwd=3,lty=2)
    
    (min.Eae <- exp(mm))
    lines(c(101,150),rep(min.Eae,2),col=SAPGold,lwd=3,lty=3)
    
    (min.Eape <- exp(mm-ss.sq))
    lines(c(101,150),rep(min.Eape,2),col=SAPGold,lwd=3,lty=4)
par(opar)

Dice rolling example:

SAPMediumGray <- "#999999"; SAPGold <- "#F0AB00"

set.seed(2013)
actuals <- sample(x=1:6,size=100,replace=TRUE)

opar <- par(mar=c(3,2,0,0)+.1)
    plot(actuals,type="o",pch=21,cex=0.8,bg="black",xlab="",ylab="",xlim=c(0,150))
    abline(v=101,col=SAPMediumGray)

    min.Ese <- 3.5
    lines(c(101,150),rep(min.Ese,2),col=SAPGold,lwd=3,lty=2)
    
    min.Eape <- 2
    lines(c(101,150),rep(min.Eape,2),col=SAPGold,lwd=3,lty=4)
par(opar)

References

Gneiting, T. Making and Evaluating Point Forecasts. Journal of the American Statistical Association, 2011, 106, 746-762

Goodwin, P. & Lawton, R. On the asymmetry of the symmetric MAPE. International Journal of Forecasting, 1999, 15, 405-408

Hoover, J. Measuring Forecast Accuracy: Omissions in Today's Forecasting Engines and Demand-Planning Software. Foresight: The International Journal of Applied Forecasting, 2006, 4, 32-35

Kolassa, S. Why the "best" point forecast depends on the error or accuracy measure (Invited commentary on the M4 forecasting competition). International Journal of Forecasting, 2020, 36(1), 208-211

Kolassa, S. & Martin, R. Percentage Errors Can Ruin Your Day (and Rolling the Dice Shows How). Foresight: The International Journal of Applied Forecasting, 2011, 23, 21-29

Kolassa, S. & Schütz, W. Advantages of the MAD/Mean ratio over the MAPE. Foresight: The International Journal of Applied Forecasting, 2007, 6, 40-43

McKenzie, J. Mean absolute percentage error and bias in economic forecasting. Economics Letters, 2011, 113, 259-262

Zheng, S. Gradient descent algorithms for quantile regression with smooth approximation. International Journal of Machine Learning and Cybernetics, 2011, 2, 191-207

$\endgroup$
10
  • 8
    $\begingroup$ Excellent Q&A. I would add that all of these metrics have two big underlying assumptions -- the series is iid and stationary. If one or both of these assumptions are not met, which happens frequently in practice, then their validity is questionable. $\endgroup$
    – user78229
    Commented Aug 25, 2017 at 12:55
  • 1
    $\begingroup$ I agree with most of this, however, wouldn't it be legitimate to deal with ratios of temperatures so long as they are on their proper scale (i.e., the Kelvin scale)? $\endgroup$
    – Ben
    Commented Jun 6, 2018 at 0:34
  • 5
    $\begingroup$ @Ben: in that case, we won't divide by zero. However, the asymmetry is still a slight problem. If your forecast is 293K and the actual is 288K, you have an APE of 1.74%, and if the forecast is 288K while the actual is 293K, the APE is 1.71%, so the second forecast looks better, though both are off by 5K. (Translate into C or F as needed.) Essentially, the same absolute errors are penalized more strongly for lower actuals. Plus, interpretation of percentage errors for temperatures is not easy. $\endgroup$ Commented Jun 6, 2018 at 6:25
  • 2
    $\begingroup$ @Ben Percentages of absolute temperature are legitimate, but differences of temperature are easier to understand - at least, when we deal with temperatures in the everyday range; when forecasting star core temperature it may be the other way. $\endgroup$
    – Pere
    Commented Sep 25, 2018 at 10:30
  • 3
    $\begingroup$ @RDizzl3. One actor's negative outflow is another actor's positive inflow, so I would have no compunctions about swapping the signs. (And MAPE still has issues, per above.) $\endgroup$ Commented Jul 26, 2021 at 19:23

Not the answer you're looking for? Browse other questions tagged or ask your own question.