3
$\begingroup$

I am working on a demand forecasting project and I am puzzled by the client's standards of forecast evaluation. The MAPE (Mean Absolute Percentage Error) with the sample data Forecast = 300 and Demand = 100 is $$ \text{MAPE} =\frac{|300-100|}{100} =2 $$

However the client focuses on forecasting accuracy. It is defined as $$ \text{Accuracy}=\max(0,1-\text{MAPE}) $$

This means that a MAPE of 1, 3 or 3000 gives the same forecasting accuracy of 0. To me this does not make sense, because it is equivalent to restricting MAPE to $\text{MAPE}_r = \max(1,\text{MAPE})$.

However, it seems to be in line with the measurement of forecasts in the demand planning ecosystem http://demandplanning.net/MAPE.htm. Can someone please explain to me why this could be useful?

EDIT: I understand that someone can define anything. My only question is whether the definition makes sense for the demand planning / management purpose.

Referring to the text in the link, restricting any error metric does not make sense to me especially(!) in demand planning. If the true demand is 1 unit but I forecasted 300, then 300 times more raw materials or human resources were planned for this product in this period. This overestimation must cause substantially higher cost than a forecast of 2 units, although both forecasts would result in a forecasting accuracy of 0. This is implied by MAPE but not by accuracy.

So why should forecasting accuracy defined as above be relevant at all? Why do I need it when MAPE is there already? What value does it add? To me it seems to introduce bias - if anything at all.

$\endgroup$
  • $\begingroup$ You had specified MAPE as mean average percentage error. The standard way of defining MAPE is mean absolute percentage error. I guess it was just a mistake, so I corrected it. $\endgroup$ – Richard Hardy Nov 28 '16 at 9:42
  • $\begingroup$ Answer to your second question is simple: you do not need it, it does not give you any additional information over MAPE, it is simply your client that prefers it for some reason. If you want to know why he prefers it, ask him... $\endgroup$ – Tim Nov 28 '16 at 10:29
  • $\begingroup$ lol thanks, but the client does not know the answer. He was told to evaluate the whole supply chain demand with this metric but cannot explain why. $\endgroup$ – HOSS_JFL Nov 28 '16 at 10:33
2
$\begingroup$

I love your quote:

He was told to evaluate the whole supply chain demand with this metric but cannot explain why.

You are completely correct that truncating "accuracy" makes no sense. It throws information away for no good reason. Much better to either accept negative "accuracy", or deal with the MAPE directly, and accept that MAPEs greater than 100% occur.

The only rationale for truncation is that there is no good interpretation of negative "accuracy". But that is a result of trying to work with "accuracy" and defining it as 1-error - where the error can be unbounded.

The following threads may be helpful:

$\endgroup$
1
$\begingroup$

Actually, this is described in the link you provided:

Error above 100% implies a zero forecast accuracy or a very inaccurate forecast. [...]

What is the impact of Large Forecast Errors?

Is Negative accuracy meaningful? Regardless of huge errors, and errors much higher than 100% of the Actuals or Forecast, we interpret accuracy a number between 0% and 100%. Either a forecast is perfect or relative accurate or inaccurate or just plain incorrect. So we constrain Accuracy to be between 0 and 100%.

Negative accuracy does not make any sense. This measure simply assumes that if something has bigger errors then the predicted value itself, then no matter how much bigger they are, they are equally bad. If you'd take a loan and then to repay it you'd have to make monthly payments that are greater then your salary, then it really does not matter how much greater they are, since you can't afford to pay them.

$\endgroup$
  • $\begingroup$ I have read the text in the link but it still does not make sense, especially in demand planning. If the true demand is 1 unit but I forecasted 300, how can this be equivalent to a forecast of 2 units? Given the fact that the production facility was expecting 300 units, raw materials were bought, people were hired etc, but only 2 units were produced? $\endgroup$ – HOSS_JFL Nov 28 '16 at 10:09
  • $\begingroup$ @HOSS_JFL so I do not understand what is your question? You are asking if your client makes a stupid decision in this scenario by choosing such measure? Maybe yes, but you did not specify that this was your question. Nobody says that each error measure must be a perfect error measure for any case. $\endgroup$ – Tim Nov 28 '16 at 10:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.