Simulate multiple forecasts with fixed accuracy

I have a time series forecast along with actual historical data, and its accuracy (MAPE, probability coverage etc.) is calculated. Now I want to estimate how improving some or all of the accuracy measures would affect a business KPI (which can be directly calculated from the forecast). I thought that I could simulate multiple forecasts with fixed accuracy (e.g. MAPE = current_MAPE - 1%) and get an empirical distribution of the KPI. What would be a proper approach to generate such forecasts?

I'm not sure what you mean by "fixed accuracy" in this context. A fixed accuracy would imply that since the accuracy is always the same (and known after the first few forecasting) we can adjust for the accuracy rate to recover the original value of the time series and we end up with a perfect forecast instead.

I assume what you really want is to simulate forecasts for which the accuracy (and hence by implication, the forecast error) is drawn from a known distribution.

In this case you have two approaches:

• (1) Assume your forecast error $$\epsilon_t$$ follows a known distribution $$P$$ (Normal, Gamma, etc...). Then do the following:

• Generate a forecast at time $$t$$, $$\hat{Y}_t$$.
• Draw a random value of $$\epsilon_t$$ from $$P$$.
• Add the error term $$\epsilon_t$$ to your forecast value $$\hat{Y}_t$$ to get a new sample of your future time series at time $$t$$
• Use that sample $$\hat{Y}_t + \epsilon_t$$ to feed it to your model and generate a forecast for the next step $$t+1$$, $$\hat{Y}_{t+1}$$.
• repeat this process: drawing a random sample from $$P$$ and then generating the next step forecast until you have reached $$\hat{Y}_{t+T}$$, the number of steps forward you want to forecast.
• Repeat the above process enough times (50, 100,...), each time using different values of $$\epsilon$$, to create enough samples from your future time series to get a good estimate of how your forecast distribution effects your KPIs and decisions.
• Alternatively, instead of assuming a distribution $$P$$ you can use the historical forecast errors (if you have already been running your model for quite a while, or through maybe through backcasting if you have enough history).
• Or (2), perform a full density forecast: This approach is more rigorous than the above mentioned approach, but is also more complicated to implement. The idea is that instead of estimating a model of your actual values $$Y_t$$, you estimate a model of the full distribution $$P(Y_t)$$ from your historical data. This can be either parametric or non-parametric. Once you have an estimate of the distribution, you can generate the sample paths directly by sampling from the distribution, as opposed to sampling from the error and then adding it back to get your sample paths. Depending on the complexity of the distribution, you can either sample from it directly, or you might have to use some MCMC method to sample from it.

• Thanks for such a thorough reply! By fixed error I meant that I'd like to generate some random forecasts (not based on a specific model, just a sequence of random values) with a constraint that average error relative to actual data would be the same, e.g. 10%. In other words, when drawing N variables from known distributions, how to ensure that these N variables satisfy some additional constraint (total error = const) Commented Jan 10, 2020 at 17:39
• @DmitryShopin you seem to be mixing two concepts. First you say "average error relative to actual data would be the same" then you say "satisfy some additional constraint (total error = const)". These are not equivalent statements: The average error can be 10% and you can still sample 9%, 8%, 11%, then 9% again, etc...but when you say errors must sum up to 10%, then once you sample 9%, you will no longer be able to sample it again, since any future values will have to be 1% or lower (e.g. your doing some sort of continuous equivalent to sampling without replacement). Commented Jan 10, 2020 at 19:57
• @DmitryShopin interesting. I'm pretty sure that you can get by with an average forecast error assumption (i.e. the first scenario) - instead of the hard constraint you are trying to impose on your error distribution. Individual error values are stochastic, but you should be able to compare the impact of an error distribution with an average error of 10% vs the impact of an error distribution with an average error of 9% without imposing any additional conditions. Commented Jan 11, 2020 at 1:15
• @DmitryShopin Note however that you are making the assumption that the cost of over-forecasting and the cost of under-forecasting are the same, but that is rarely the case in real world business scenarios in my experience. Usually they are very different, and you should way your forecast errors differently (maybe you can do that with quantile forecasts?) or better still, look at analyzing and optimizing the decision process that consumes the forecast, and then work backwards from there. Commented Jan 11, 2020 at 1:17
• Here is an example from my world of inventory and demand forecasting: lokad.com/accuracy-gains-(inventory) Commented Jan 11, 2020 at 1:19