The set up:

Me and a friend set up a website which employees of a small company (<10 people) submit forecasts for the next month's revenue of different sections of the business, and also an estimate for the business as a whole. These estimates consist of them assigning a probability to several bins. (e.g., for a branch of the business based in California, they might forecast a 15% chance of monthly revenue being below £10k, and 40% chance of it being between £10-15k, a 30% chance of being £15-20k and a 15% chance of being over £20k)

We then can measure retrospectively how good these forecasts are, e.g. with a brier score or similar.

The problem:

We then want to do some inference. Given the forecasts for the next month, produce a sensible estimate. My current thinking is as follows:

  • For a point estimate, we could take a simple average, perhaps weighted by past accuracy (although so far we only have one month's data).
  • For an error estimate, we would probably want to assume the underlying forecast of each individual follows a distribution, that individual forecasts are independent, and then aggregate them. E.g. for estimating the revenue earned by the Californian branch in the example above, they are estimating a vector of probabilities, and we have some measure of how correct or incorrect they were in the past

For this second part, I am not sure what distribution assumption would be appropriate - I have only come across a limited class of parametric distributions in the statistics I have covered so far, or what might be a sensible way to update the weights with more forecasts.

Secondly, at first we will only have a very small set of data, although that will be increasing over time.

(For reference: my background is more in mathematics than in statistics, but have taken an elementary statistics course, largely in normal linear models, and some hypothesis testing. It is only in the third year of our course that we take less elementary statistics courses)

Edit: In light of Stephen Kolassa's comment, instead of a distribution overlay, what might be better is a reasonable metric of the uncertainty of the forecast, rather than some meta distribution of the forecast

  • 1
    $\begingroup$ If your "input" forecasts are binned, then the simplest way to obtain a consensus density forecast would be to just use the same bins. I would be wary of overlaying a distribution, which smacks of conjuring information out of thin air. Am I misunderstanding? $\endgroup$ – Stephan Kolassa Apr 15 at 15:56
  • $\begingroup$ so that was my initial thought. However, I would also like to get some measure of the uncertainty of the forecast? (which will hopefully decrease with more data!) $\endgroup$ – Ethan Horsfall Apr 15 at 15:58
  • 1
    $\begingroup$ You mean the uncertainty of a probabilistic forecast? That could in principle be done, but it does sound a little meta to me (and probably incomprehensible to most consumers of forecasts, who typically already have a hard time understanding density forecasts as such). $\endgroup$ – Stephan Kolassa Apr 15 at 16:00
  • 1
    $\begingroup$ Hm. I'm all for helping people see how they are improving things. I would say that your best bet would be to show how the average expectation forecast will (very probably) be better than each separate expectation forecast. I would also assume the combined density forecast to be better than the separate ones, but even assessing density forecasts (via proper scoring rules) is very unintuitive, and any improvement in a score will be hard for people to wrap their heads around. $\endgroup$ – Stephan Kolassa Apr 15 at 16:10
  • 2
    $\begingroup$ Incidentally, weighting by past performance may improve the consensus forecast. Or it may not. See Claeskens et al. (2016, IJF). I would very much recommend you compare your weighted combination to an unweighted one. $\endgroup$ – Stephan Kolassa Apr 15 at 16:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.