3
$\begingroup$

Let say I have a forecasting system compared with a naive forecast that just use the today's value as forecast. If the naive forecast have a MAPE of 200%, and my system have a MPAE of 100%, could I say my system is useful?

On one hand, my system halved the MAPE, but on the other hand, my system still have a 100% MAPE, while many text book, guideline suggested that MAPE should be <10%. but some guideline say it is still depending on the performance of baseline model.

Other metric like RMSE and MAE also was halved by my system.

Any comments and suggestions are appreciated.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
8
$\begingroup$

Let say I have a forecasting system compared with a naive forecast that just use the today's value as forecast. If the naive forecast have a MAPE of 200%, and my system have a MAPE of 100%, could I say my system is useful?

That can be useful, or not. On the one hand, smaller errors are always more impressive. On the other hand, the relationship between forecast accuracy and value is absolutely not straightforward (Kolassa, 2023). On the third hand, the MAPE especially has major issues: What are the shortcomings of the Mean Absolute Percentage Error (MAPE)? A flat zero forecast will give you a MAPE of 100%, but whether this is in any way more useful than a non-zero forecast with a MAPE of 200% is still very doubtful.

Assess the value of your forecast, not the error.

many text book, guideline suggested that MAPE should be <10%.

It makes no sense whatsoever to give a hard number as to what an error "should" be (other than a MAPE greater than 100% can always be beaten by a flat zero forecast). The possible forecast accuracy is always context dependent. If you forecast a coin toss with equally probable outcomes of zero and one, then you will not be able to get a better weighted MAPE than 100%. the score to hope for when evaluating model by MAE, MSE or RMSE and Kolassa (2008).

If your textbook tells you that the MAPE "should" be less than 10% without reference to a very specific dataset, then you should get rid of that textbook. There are better books out there, many of them online for free: Resources/books for project on forecasting models

Improve this answer
$\endgroup$
2
  • $\begingroup$ I think the RMSE and MAE also halved suggesting that in term of forecast value, my system is also better. I do see that many forecasting paper in my field did not give MAPE instead, so is it something that we should always use? $\endgroup$
    – Mahali Sindy
    Commented Dec 11, 2023 at 9:16
  • $\begingroup$ Reductions in other metrics also sounds good. I don't quite understand your second question, are you asking whether you should use the MAPE more? I would not say so, I believe the MAPE is dangerous and misleading. See the special feature in here, ping me via ResearchGate for my simulation article if you are interested, and I can send you the entire special feature. $\endgroup$
    – Stephan Kolassa
    Commented Dec 11, 2023 at 9:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.