I am working on a financial model that will forecast the revenue a company generates over a fiscal quarter, and I am not sure of the best way to rigorously evaluate the bias in the model.

Every day throughout the quarter I generate a new forecast for what revenue will be at the end of the quarter. Just for simplicity, let's assume that the error estimate on each forecast is specified by the standard-deviation of a normal distribution. As I approach the end of the quarter, the size of my error estimate will decrease until the final day when my forecast should agree with what actually occurred.

Once the quarter is complete, I can evaluate how far off each day's estimate was from what actually occurred. My question is this: How can I combine the estimate, uncertainty for each day with the final outcome to provide a metric of model quality and/or bias? Just for context, the forecast for each day is generated by Monte-Carlo simulation over the relevant state of the current/historical sales process as it existed on that day.

This question is illustrated by the accompanying figure (revenue on y-axis). The black dots are point estimates of quarterly revenue. The blue shaded area represents the uncertainty in each estimate. The heavy blue line represents the cumulative revenue generated for the quarter. The dashed black line represents the final outcome for total revenue that quarter. enter image description here

--edit-- For clarification, I basically want to answer the question. "How good is this model." Maybe "bias" isn't the best metric for that. At the end of the day, I just want the forecast distribution for any day to match the distribution of actual outcomes.

  • $\begingroup$ How about a usual metric such as RMSE? $\endgroup$ – user2974951 Jul 22 at 6:28
  • $\begingroup$ Does that work as expected when the errors at different times are highly correlated like this? If I had many replicates of this timeseries, then I can see how to do it. But checking for "model goodness" with only one time series is what I'm after. There may not be a good way to do it(?) $\endgroup$ – Rob deCarvalho Jul 23 at 14:09
  • $\begingroup$ I don't see a problem with it. For every prediction you subtract the final actual (true) value and calculate the RMSE or something similar. This will give you a measure of spread of your predictions, presumably you want a model that has very little spread, one that predicted values close to the true value from the start. Correlation is not an issue. Also I wouldn't talk about bias but simply variance. $\endgroup$ – user2974951 Jul 24 at 11:48
  • $\begingroup$ Agreed on the "not talking about bias." I should not have framed the question that way. I'll think some more about your RMSE solution. Somehow this problem seems related to putting error bars on an empirical cumulative distribution function -- which I also do not know how to do correctly. $\endgroup$ – Rob deCarvalho Jul 25 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.