1
$\begingroup$

I am building a model for forecasting some number of activations. My data set has a panel structure.

Now, I want to come up with a forecast performance measure to assess the performance of my model and to compare it with other models. In this measure I want to punish too low forecasts differently from too high forecasts (so an asymmetric performance measure). I was thinking about the MAPE, but my data also contains 0 (and small) values, so the MAPE will not work. As an extra I would like to be able to change the degree to which too high predictions are punished differently from too low predictions.

I want the asymmetric measure because too high forecasts are worse than too low forecasts, since too high predictions leads to unused stock while too low stock allows for acquiring new stock which is less severe in my case.

Does anyone have a suggestion which performance measure I could use?

$\endgroup$
  • $\begingroup$ What is the reason you want to penalize underforecasts differently than overforecasts? $\endgroup$ – Stephan Kolassa Jan 23 at 10:46
  • $\begingroup$ Edit: I want the asymmetric measure because too high forecasts are worse than too low forecasts, since too high predictions leads to unused stock while too low stock allows for acquiring new stock which is less severe in my case. $\endgroup$ – user235042 Jan 23 at 10:53
  • $\begingroup$ You should define then a cost function that takes that into account. If you do that I highly recommend to define the actual (it seems that is an economic cost) into which you incurr. $\endgroup$ – chuse Jan 23 at 11:00
0
$\begingroup$

If you are forecasting for inventory control, then you actually want a (high) quantile forecast. That is, you need a quantity that exceeds the actual demand in 90% or 95% of all periods.

Your forecasting tool almost certainly does not aim at or output a quantile forecast, but a forecast of expected demand. Differentially penalizing over-/underforecasts of this expectation forecast will not get you to a good quantile forecast.

I discuss some aspects of this in more detail here. That answer also includes, at the very bottom, a link to possible error measures for quantile forecasts, which you may be interested in. Just make sure you don't apply them to an expectation forecast, which makes no sense.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.