I am working on a financial model that will forecast the revenue a company generates over a fiscal quarter, and I am not sure of the best way to rigorously evaluate the bias in the model.
Every day throughout the quarter I generate a new forecast for what revenue will be at the end of the quarter. Just for simplicity, let's assume that the error estimate on each forecast is specified by the standard-deviation of a normal distribution. As I approach the end of the quarter, the size of my error estimate will decrease until the final day when my forecast should agree with what actually occurred.
Once the quarter is complete, I can evaluate how far off each day's estimate was from what actually occurred. My question is this: How can I combine the estimate, uncertainty for each day with the final outcome to provide a metric of model quality and/or bias? Just for context, the forecast for each day is generated by Monte-Carlo simulation over the relevant state of the current/historical sales process as it existed on that day.
This question is illustrated by the accompanying figure (revenue on y-axis). The black dots are point estimates of quarterly revenue. The blue shaded area represents the uncertainty in each estimate. The heavy blue line represents the cumulative revenue generated for the quarter. The dashed black line represents the final outcome for total revenue that quarter.
--edit-- For clarification, I basically want to answer the question. "How good is this model." Maybe "bias" isn't the best metric for that. At the end of the day, I just want the forecast distribution for any day to match the distribution of actual outcomes.