The set up:
Me and a friend set up a website which employees of a small company (<10 people) submit forecasts for the next month's revenue of different sections of the business, and also an estimate for the business as a whole. These estimates consist of them assigning a probability to several bins. (e.g., for a branch of the business based in California, they might forecast a 15% chance of monthly revenue being below £10k, and 40% chance of it being between £10-15k, a 30% chance of being £15-20k and a 15% chance of being over £20k)
We then can measure retrospectively how good these forecasts are, e.g. with a brier score or similar.
We then want to do some inference. Given the forecasts for the next month, produce a sensible estimate. My current thinking is as follows:
- For a point estimate, we could take a simple average, perhaps weighted by past accuracy (although so far we only have one month's data).
- For an error estimate, we would probably want to assume the underlying forecast of each individual follows a distribution, that individual forecasts are independent, and then aggregate them. E.g. for estimating the revenue earned by the Californian branch in the example above, they are estimating a vector of probabilities, and we have some measure of how correct or incorrect they were in the past
For this second part, I am not sure what distribution assumption would be appropriate - I have only come across a limited class of parametric distributions in the statistics I have covered so far, or what might be a sensible way to update the weights with more forecasts.
Secondly, at first we will only have a very small set of data, although that will be increasing over time.
(For reference: my background is more in mathematics than in statistics, but have taken an elementary statistics course, largely in normal linear models, and some hypothesis testing. It is only in the third year of our course that we take less elementary statistics courses)
Edit: In light of Stephen Kolassa's comment, instead of a distribution overlay, what might be better is a reasonable metric of the uncertainty of the forecast, rather than some meta distribution of the forecast