How do I decide when to use MAPE, SMAPE and MASE for time series analysis on stock forecasting

My task is to forecast future 1 month stock required for retail store, at a daily basis. How do I decide whether MAPE, SMAPE and MASE is a good metrics for the scenario?

In my context, over-forecast is better than under-forecast.

You are forecasting for stock control, so you need to think about setting safety amounts. In my opinion, a quantile forecast is far more important in this situation than a forecast of some central tendency (which the accuracy KPIs you mention assess).

You essentially have two or three possibilities.

1. Directly forecast high quantiles of your unknown future distribution. There are more and more papers on this. I'll attach some below.

Regarding your question, you can assess the quality of quantile forecasts using hinge loss functions, which are also used in quantile regression. Take a look at the papers by Ehm et al. (2016) and Gneiting (2011) below.

2. Forecast some central tendency, e.g., the conditional expectation, plus higher moments as necessary, and combine these with an appropriate distributional assumption to obtain quantiles or safety amounts. For instance, you could forecast the conditional mean and the conditional variance and use a normal or negative-binomial distribution to set target service levels.

In this case, you can use a forecast accuracy KPI that is consistent with the measure of central tendency you are forecasting for. For instance, if you try to forecast the conditional expectation, you can assess it using the MSE. Or you could forecast the conditional median and assess this using the MAE, wMAPE or MASE. See Kolassa (2019) on why this sounds so complicated. And you will still need to assess whether your forecasts of higher moments (e.g., the variance) are correct. Probably best to directly evaluate the quantiles this approach yields by the methods discussed above.

3. Forecast full predictive densities, from which you can derive all quantiles you need. This is what I argue for in Kolassa (2016).

You can evaluate predictive densities using proper scoring rules. See Kolassa (2016) for details and pointers to literature. The problem is that these are far less intuitive than the point forecast error measures discussed above.

What are the shortcomings of the Mean Absolute Percentage Error (MAPE)? is likely helpful, and also contains more information. If you are forecasting for a single store, I suspect that the MAPE will often be undefined, because of zero demands (that you would need to divide by).

References

(sorry for not nicely formatting these)

Ehm, W.; Gneiting, T.; Jordan, A. & Krüger, F. Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings (with discussion). Journal of the Royal Statistical Society, Series B, 2016 , 78 , 505-562

Gneiting, T. Quantiles as optimal point forecasts. International Journal of Forecasting, 2011 , 27 , 197-207

Kolassa, S. Why the "best" point forecast depends on the error or accuracy measure. International Journal of Forecasting, 2020 , 36, 208-211

Kolassa, S. Evaluating Predictive Count Data Distributions in Retail Sales Forecasting. International Journal of Forecasting, 2016 , 32 , 788-803

The following are more generally on quantile forecasting:

Trapero, J. R.; Cardós, M. & Kourentzes, N. Quantile forecast optimal combination to enhance safety stock estimation. International Journal of Forecasting, 2019 , 35 , 239-250

Bruzda, J. Quantile smoothing in supply chain and logistics forecasting. International Journal of Production Economics, 2019 , 208 , 122 - 139

Kourentzes, N.; Trapero, J. R. & Barrow, D. K. Optimising forecasting models for inventory planning. Lancaster University Management School, Lancaster University Management School, 2019

Ulrich, M.; Jahnke, H.; Langrock, R.; Pesch, R. & Senge, R. Distributional regression for demand forecasting -- a case study. 2018

Bruzda, J. Multistep quantile forecasts for supply chain and logistics operations: bootstrapping, the GARCH model and quantile regression based approaches. Central European Journal of Operations Research, 2018

• I read somewhere that setting a Service Level was essentially equivalent to a quantile forecast: While that makes sense intuitively, the exact math is little bit fuzzy for me: How does this tie to the OP? Is a service level better expressed in terms of MAPE? RMSE? etc.... Sep 9, 2019 at 21:26
• @SkanderH.: good question. This actually depends heavily on what definition of "service level" you use. (Yes, there are multiple ones. Some have a straightforward connection to quantiles, most don't.) If you are deeply interested, do ask a question here or at OR.SE and ping me, but I can't promise I'll get around to an answer soon - it would be a longer one, and I am a bit frazzled right now. Sep 10, 2019 at 6:52
• MAPE, RMSE etc. have nothing to do with service levels. Nothing whatsoever. "Nothing" as in "should I wear red or green while rewiring my house". These KPIs measure the agreement of forecasts with (different!) central tendencies of the future demand distribution, but service levels don't care about central tendencies at all, but only about quantiles (and per above, the relationship is not necessarily straightforward). Sep 10, 2019 at 6:54
• So there is an OR SE...and there I was posting all of my optimization questions in the math SE, and wondering why the responses were so sparse.... Sep 10, 2019 at 18:11