12
$\begingroup$

I have a time series that deals with rainfall. It is a period of 10 years (daily resolution), and covers climate variables.

I'm going to feed the data into an Artificial Neural Network to predict the rainfall variable (PP).

As what I've been reading, MAPE's formula involves dividing by the actual observed value. But since its rainfall, there will be days with little or zero precipitation values.

This is bad (dividing by zero = black hole). So how am I going to go about this? I could do data replacement on the zero or close to zero values, but that's stupid - if I do that, I inflate a lot of things, and am pretty much tampering with the data in a way (unlike missing values, which should be imputed by way of other data and not filled in with some other arbitrary value).

My professor is stubborn as a mule. Is there any alternative to MAPE? Or are there any methods to circumvent the issues of MAPE?

EDIT

THERE ARE SMALL AND ZERO VALUES IN THE DATASET... Am I just screwed now?

Improve this question
$\endgroup$
5
  • $\begingroup$ Check out F.X.Diebold's free textbook "Forecasting in Economics, Business, Finance and Beyond", Chapter 10 "Point forecast evaluation". You will find mean squared error, mean absolute error, predictive $R^2$ and Theil's $U$ statistic. Another measure could be mean absolute scaled error. $\endgroup$
    – Richard Hardy
    Commented May 19, 2017 at 8:36
  • $\begingroup$ Is there really no other circumvention that can lead me to still use MAPE? I know of these alternatives, I've read up on a few of them. But I wanted an insider's opinion on this matter. Am I just gonna have to give my professor the bird and use another error measurement? $\endgroup$
    – ace_01S
    Commented May 19, 2017 at 8:39
  • $\begingroup$ Look at Tim's answer and show the references to your professor. Hopefully that will be convincing enough. If not, ask him how he thinks MAPE should be calculated when the true values are zeros. $\endgroup$
    – Richard Hardy
    Commented May 19, 2017 at 8:57
  • 1
    $\begingroup$ See en.wikipedia.org/wiki/Mean_absolute_scaled_error $\endgroup$
    – Rob Hyndman
    Commented May 19, 2017 at 9:55
  • $\begingroup$ Everybody seems to know it, I didn't: MAPE stands for mean absolute percentage error $\endgroup$
    – normanius
    Commented Feb 13, 2020 at 23:04

1 Answer 1

Reset to default
20
$\begingroup$

No, actually MAPE is very poor error measure as discussed by Stephan Kolassa in Best way to optimize MAPE and Prediction Accuracy - Another Measurement than MAPE and Minimizing symmetric mean absolute percentage error (SMAPE) and on those slides. You can also check the following paper:

Tofallis, C. (2015). A better measure of relative prediction accuracy for model selection and model estimation. Journal of the Operational Research Society, 66(8), 1352-1362.

It is also discussed by Goodwin and Lawton (1999) in the On the asymmetry of the symmetric MAPE paper

Despite its widespread use, the MAPE has several disadvantages (Armstrong & Collopy, 1992; Makridakis, 1993). In particular, Makridakis has argued that the MAPE is asymmetric in that ‘equal errors above the actual value result in a greater APE than those below the actual value’. Similarly, Armstrong and Collopy argued that ‘the MAPE ... puts a heavier penalty on forecasts that exceed the actual than those that are less than the actual. For example, the MAPE is bounded on the low side by an error of 100%, but there is no bound on the high side’.

The quoted (Makridakis, 1993) paper gives a nice example for the asymmetry, when the predicted value is $150$ and the forecast is $100$, MAPE is $|\tfrac{150-100}{150}| = 33.33\%$, while when the predicted value is $100$ and the forecast is $150$ MAPE is $|\tfrac{100-150}{100}| = 50\%$ despite the fact that both forecasts are wrong by $50$ units!

What the above references, and the number of other sources, show is that if you use MAPE as a criterion for selecting your forecasts, this would lead to biased and underestimated results. Moreover you run into problems when the predicted value is equal to zero.

In the How to interpret error measures in Weka output? thread you can find a brief review of other error measures.

Improve this answer
$\endgroup$
13
  • $\begingroup$ Is there a close enough catch-all measurement statistic that can be used for pretty much any scenario? By extension, he's pretty adamant on anything that has "percentage" in it, saying that it sinks in easier to people if we're given a percentage (and I agree with this). Thoughts? $\endgroup$
    – ace_01S
    Commented May 19, 2017 at 9:10
  • 3
    $\begingroup$ @Ace_01S no there is not. The choice of error measure is always problem specific. $\endgroup$
    – Tim
    Commented May 19, 2017 at 9:25
  • 1
    $\begingroup$ @Tim, can you elaborate more, please? Yes, both forecasts in your example are wrong by 50 units, by being wrong by 50 units is obviously a bigger error for a target of 100 vs. 150. Isn't it? $\endgroup$
    – towi_parallelism
    Commented Aug 6, 2019 at 11:42
  • 1
    $\begingroup$ It is simple math, but results in two very different interpretations. You are saying treating 50 units off from 100 and 50 units off from 150 differently is an example of asymmetry caused by MAPE. I am saying the example is asymmetric in its nature. being 50 off from the baseline of 100 is totally different from being 50 off from the baseline of 150. Maybe we just need a better example $\endgroup$
    – towi_parallelism
    Commented Aug 6, 2019 at 15:40
  • 1
    $\begingroup$ From what I could gather, what is hidden under a rather vague "asymmetry" label, is NOT what it appears to be. It is correct that the over-forecast and under-forecast by the same amount results in the same MAPE value for a fixed actual. However, imagine a situation where the actual can be 1 or 3 with 1/2 probability each. In this case, the forecast which minimises the expected MAPE is 1.5 rather than a seemingly more obvious 2, hence the "asymmetry" label. Explained by Kolassa: stats.stackexchange.com/questions/299712/… $\endgroup$
    – shiftyscales
    Commented Dec 23, 2022 at 14:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.