2
$\begingroup$

I have a time series that deals with rainfall. It is a period of 10 years (daily resolution), and covers climate variables.

I'm going to feed the data into an Artificial Neural Network to predict the rainfall variable (PP).

As what I've been reading, MAPE's formula involves dividing by the actual observed value. But since its rainfall, there will be days with little or zero precipitation values.

This is bad (dividing by zero = black hole). So how am I going to go about this? I could do data replacement on the zero or close to zero values, but that's stupid - if I do that, I inflate a lot of things, and am pretty much tampering with the data in a way (unlike missing values, which should be imputed by way of other data and not filled in with some other arbitrary value).

My professor is stubborn as a mule. Is there any alternative to MAPE? Or are there any methods to circumvent the issues of MAPE?

EDIT

THERE ARE SMALL AND ZERO VALUES IN THE DATASET... Am I just screwed now?

$\endgroup$
  • $\begingroup$ Check out F.X.Diebold's free textbook "Forecasting in Economics, Business, Finance and Beyond", Chapter 10 "Point forecast evaluation". You will find mean squared error, mean absolute error, predictive $R^2$ and Theil's $U$ statistic. Another measure could be mean absolute scaled error. $\endgroup$ – Richard Hardy May 19 '17 at 8:36
  • $\begingroup$ Is there really no other circumvention that can lead me to still use MAPE? I know of these alternatives, I've read up on a few of them. But I wanted an insider's opinion on this matter. Am I just gonna have to give my professor the bird and use another error measurement? $\endgroup$ – ace_01S May 19 '17 at 8:39
  • $\begingroup$ Look at Tim's answer and show the references to your professor. Hopefully that will be convincing enough. If not, ask him how he thinks MAPE should be calculated when the true values are zeros. $\endgroup$ – Richard Hardy May 19 '17 at 8:57
  • $\begingroup$ See en.wikipedia.org/wiki/Mean_absolute_scaled_error $\endgroup$ – Rob Hyndman May 19 '17 at 9:55
  • $\begingroup$ Everybody seems to know it, I didn't: MAPE stands for mean absolute percentage error $\endgroup$ – normanius Feb 13 at 23:04
6
$\begingroup$

No, actually MAPE is very poor error measure as discussed by Stephan Kolassa in Best way to optimize MAPE and Prediction Accuracy - Another Measurement than MAPE and Minimizing symmetric mean absolute percentage error (SMAPE) and on those slides. You can also check the following paper:

Tofallis, C. (2015). A better measure of relative prediction accuracy for model selection and model estimation. Journal of the Operational Research Society, 66(8), 1352-1362.

It is also discussed by Goodwin and Lawton (1999) in the On the asymmetry of the symmetric MAPE paper

Despite its widespread use, the MAPE has several disadvantages (Armstrong & Collopy, 1992; Makridakis, 1993). In particular, Makridakis has argued that the MAPE is asymmetric in that ‘equal errors above the actual value result in a greater APE than those below the actual value’. Similarly, Armstrong and Collopy argued that ‘the MAPE ... puts a heavier penalty on forecasts that exceed the actual than those that are less than the actual. For example, the MAPE is bounded on the low side by an error of 100%, but there is no bound on the high side’.

The quoted (Makridakis, 1993) paper gives a nice example for the asymmetry, when the predicted value is $150$ and the forecast is $100$, MAPE is $|\tfrac{150-100}{150}| = 33.33\%$, while when the predicted value is $100$ and the forecast is $150$ MAPE is $|\tfrac{100-150}{100}| = 50\%$ despite the fact that both forecasts are wrong by $50$ units!

What the above references, and the number of other sources, show is that if you use MAPE as a criterion for selecting your forecasts, this would lead to biased and underestimated results. Moreover you run into problems when the predicted value is equal to zero.

In the How to interpret error measures in Weka output? thread you can find a brief review of other error measures.

$\endgroup$
  • $\begingroup$ Is there a close enough catch-all measurement statistic that can be used for pretty much any scenario? By extension, he's pretty adamant on anything that has "percentage" in it, saying that it sinks in easier to people if we're given a percentage (and I agree with this). Thoughts? $\endgroup$ – ace_01S May 19 '17 at 9:10
  • 2
    $\begingroup$ @Ace_01S no there is not. The choice of error measure is always problem specific. $\endgroup$ – Tim May 19 '17 at 9:25
  • $\begingroup$ @Tim, can you elaborate more, please? Yes, both forecasts in your example are wrong by 50 units, by being wrong by 50 units is obviously a bigger error for a target of 100 vs. 150. Isn't it? $\endgroup$ – towi_parallelism Aug 6 '19 at 11:42
  • $\begingroup$ @towi_parallelism "bigger" in what sense? Both are wrong by the same quantity. $\endgroup$ – Tim Aug 6 '19 at 13:16
  • $\begingroup$ @Tim, yes, but being wrong by 50 when your actual value is 100 is a bigger error than being wrong by 50 when your actual value is 1 million. Isn't it? That's what is happening in the example too (actual values in your examples are different, hence the MAPE) $\endgroup$ – towi_parallelism Aug 6 '19 at 13:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.