1
$\begingroup$

I'm doing a weekly time series analysis and I'm generally getting a mape of 35% is that ok according to the industry standard?

$\endgroup$
6
$\begingroup$

There is no such standard and there never will be. If your data is very noisy and forecasting-unfriendly, you will be happy to find any forecasting method that behaves better then the best alternative you have for it. There is no arbitrary "critical" value that marks the point where you can say that your forecast is either "good", or "bad". If you decided to adopt such rule-of-thumb, then unavoidably you will end up with concluding that for some data no matter what method you will use, all of them will produce "good" forecasts, and for other data, you will be unable to find any method that produces one.

See also the How to know that your machine learning problem is hopeless? thread.

$\endgroup$
  • $\begingroup$ Thank you, I understood your point but isn't there any acceptable cut off which could be taken? I mean if I'm getting a MAPE up to 10% I would generally trust the forecast. But can I accept the forecast when it gives the MAPE of around 30+? I just have over 2 years of weekly data. As my data is also less could be leading towards high MAPE $\endgroup$ – Shivam Sarin Apr 18 '17 at 11:49
  • 3
    $\begingroup$ @ShivamSarin As stated above, there cannot be any such level that works for every series and for every decision-making process based on a forecast. If you feel that, through your experience and domain knowledge related to a specific series, up to 10% is fine, then there's your answer. In general, models can be compared to each other and ranked, but ultimately what is acceptable is up to the person who takes decisions based on the forecast (and their utility function!). $\endgroup$ – Chris Haug Apr 18 '17 at 12:10
1
$\begingroup$

+1 to Tim's answer.

There are quite a number of published "benchmarks" of forecast accuracy out there, but in my opinion, they are all misleading. Typically, they will be based on responses to informal surveys at practitioner forecasting conferences and similar, and if you think about things for a little, you will notice that industries are widely different, time series are widely different (high noise vs. low noise, strong seasonality vs. no seasonality, promotional or other external drivers or not, ...) and so forth. Plus people will even differ on how they calculate the MAPE (Green & Tashman, 2009. Percentage Error: What Denominator? Foresight: The International Journal of Applied Forecasting, 12, 36-40).

These differences mean that there simply cannot be useful external benchmarks for forecasting accuracy. I summarized my arguments in a little paper (Kolassa, 2008. Can We Obtain Valid Benchmarks from Published Surveys of Forecast Accuracy? Foresight: The International Journal of Applied Forecasting, 11, 6-14) which may be helpful.

Instead, I'd recommend that you look at internal benchmarks: throw different forecasting algorithms at your data and see how they compare. Consider averaging different forecasts, which often improves accuracy. Think about whether you have modeled all important drivers, like trend, seasonality or promotions. (Conversely, don't model weak effects, which may make your forecasts worse.) Look at badly forecasted series: why are they badly forecasted? Do they exhibit some kind of unmodeled regularity?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.