Best way to optimize MAPE

Question

The MAPE is a metric that can be used for regression problems :

$$\mbox{MAPE} = \frac{1}{n}\sum_{t=1}^n \left|\frac{A_t-F_t}{A_t}\right|$$

Where $A$ represents the actual value and $F$ the the forecast.

I have to optimize my models with respect to this metric. However, I am not sure of the best way to proceed. I could rewrite the objective function of my models (but most of the common libraries do not support custom objective functions) but this requires a lot of efforts.

Alternatively, I could use a transformation $f$ of the target, run the learning on the image $f(target)$ and return the prediction $f^{-1}(predicted)$.

I noticed that training the model, keeping the sum of squares metric, on $\log(target)$ and returning $\exp(predicted)$ resulted in a significant improvement.

Is there a way to know the best transform to use? Or should I cross validate over various transformations of the target?

Answer 1

I noticed that training the model, keeping the sum of squares metric, on log(target) and returning exp(predicted) resulted in a significant improvement.

This does not surprise me. Look at things probabilistically. Your out-of-sample targets follow a certain unknown distribution. You are calculating a point forecast, which is a one-point summary of this unknown distribution, using the expected MAPE as a loss function.

The first key question is the following: which functional of the unknown future distribution minimizes the expected MAPE? Minimizing the MAPE will automatically draw your predictions to this functional. It turns out that the expected MAPE is minimized by a somewhat exotic functional, the (-1)-median of the future distribution (Gneiting, 2011, JASA, p. 752 with $\beta = -1$).

However, and this is the second key question, I strongly suspect that your estimation procedure does not aim at this functional, but uses a loss function that is minimized by the expected value of the residuals. This discrepancy between the loss function used by your model estimation algorithm and the loss function you use to evaluate predictions leads to strange results.

The MAPE tag wiki contains pointers to literature.

Here is an example. Suppose you have lognormally distributed data, with mean 0 and variance 1 on the log scale, and you have no useful predictors. Without predictors, the fit should be the same for all observations.

• Minimizing the in-sample squared error will lead to a fit which is simply the mean of your data, or $\sqrt{e}\approx 1.65$. A simulation tells us that this fit yields an expected MAPE of about 197% and an expected MSE of about 4.70.
• However, minimizing the MAPE will lead to the (-1)-median of your data, which for the lognormal distribution turns out to be its mode at $\frac{1}{e}\approx 0.37$. The minimal expected MAPE is 68%, while the expected MSE for this fit is 6.34.

The MSE-optimal fit is five times as large as the MAPE-optimal fit, because of the asymmetry of the MAPE - fits that are too large can incur APEs larger than 100%, while the APE is bounded by 100% for fits that are too small. This pulls the MAPE-optimal fit towards zero.

• If you take logarithms of your data and then estimate an MSE-minimizing fit on the logged data, we of course get the mean of the logged data, which is zero. Backtransforming this yields a fit of 1, which is optimal for neither the MAPE nor the MSE for original data, but is closer to the MAPE-optimal value than the original MSE-optimal fit. It yields a MAPE of 113% and an MSE of 5.12.

You probably have asymmetrically distributed data. I would take a very critical look at whether the MAPE is really a useful error measure in such a situation. If you do decide to minimize the MAPE, the best solution would quite probably indeed be to change the objective function. If this is not possible, cross-validation and checking various parameters for (say) Box-Cox transformations may be your best bet. (But it would still feel like hammering a nail into a board using a screwdriver instead of a hammer.) In addition, I would very much recommend that you look at your fits' and forecasts' bias and think about whether large biases are a source of concern.

Here is R code for the simulations:

set.seed(1)
xx <- rlnorm(1e7)
yy <- sqrt(exp(1))
yy
mean(abs(xx-yy)/xx)
mean((xx-yy)^2)

yy <- 1/exp(1)
yy
mean(abs(xx-yy)/xx)
mean((xx-yy)^2)

yy <- 1
mean(abs(xx-yy)/xx)
mean((xx-yy)^2)

Answer 2

Estimating or selecting a model based on its goodness of fit, predictive performance, classification performance, and other fitting-related quantities is not recommended since this strategy tends to over-fit. The best model, based on these criteria, will always be the model with more parameters, since they are always more flexible and, consequently, provide a better fit and better (if little in many cases) predictive performance. The price to pay: using more parameters than you need and getting more spread confidence intervals, and all the unpleasant consequences of over-fitting.