Combining expert estimates

Question

I am trying to work on a process to improve how how well my team estimates. I want to look at using some statistics to help out and embrace the uncertainty in how we estimate tasks.

If I have a group of experts and I ask them for a best (b), average (a) and worst (w) case estimate for a task, creating a triangular distribution.

Using a monte carlo method I can generate 10,000 trials using the triangular distribution for each of the estimators and combine these together into one sample set.

I can then use a MLE to create a beta distribution from the sample statistics, then use the cdf(x) functions to determine the likelihood of an estimated number of days, or use the inverseF to determine the number of days to maintain a certain likelihood.

This seems to make sense to me logically, however I am unsure about the fitting of a Beta distribution to the samples. I am wondering if there is a better distribution I should be using, especially given that the samples are just randomly generated from N triangular distributions.

Does anybody have any suggestions or experience in doing this that could point me in the right direction? It's been a few years since I left uni, so my statistics is a little rusty. There are plenty of examples of this process for one expert, however I would like to be able to do the same thing for multiple experts.

Thank you,

Aidan

Answer 1

I don't know how to answer your question. However, I want to give one slight word of caution about trusting the "worst case" and "best case" estimates too much.

There has been a lot of research suggesting that people tend to be overly confident in their estimates, and to underestimate the degree of uncertainty in their answers. For instance, you can give people a 20-question quiz, where each quiz question asks them something they don't know (e.g., "How many gas stations are there in the US?"), but rather than asking them to give the answer, you ask them to give a 90% confidence interval. Then, you grade their quiz by marking their answer correct if the true answer falls within their chosen interval, and false otherwise. If we were perfect at estimating our own level of confidence accurately, you would expect that people would score on average about 18 out of 20 questions correct. Instead, what you find is that most people's scores are much lower. In other words, they choose a confidence interval that is too narrow, underestimating their lack of knowledge or the amount of uncertainty in their estimate.

Therefore, if you ask people for the best case and worst case time to complete a task, I would expect their worst-case estimates to be too optimistic: I would expect that the true worst case may be significantly worse than their estimates would indicate. This bias seems to be a fundamental human phenomenom, that is difficult to counter (even for experts). So, just beware that the amount of uncertainty in your estimates may be greater than your polls of the experts may indicate. I don't know of any easy way to correct for this through statistics; it is just something you'll have to be aware of and keep in mind.

Answer 2

I think you're being a little over-ambitious --- using the beta distribution vs. another distribution function is the least of your problems. I'm not really sure what resampling the forecasts is supposed to achieve, and then re-estimating some parameters based on those resampled data isn't getting you anything that couldn't be estimated from the original forecasts.

A better approach might be to track the estimate and the actual outcome over time and keep track of some other information about the task. Then you can look at bias (i.e. the average difference between the estimate and the outcome) and how the bias depends on other task characteristics. Unless you're in a setting where you have a lot of historical data (offhand, I'd say estimates and outcomes for ~ 20 past tasks if you don't car about conditioning on characteristics; more if you do, but that number's just off the top of my head) or are going to be accumulating new forecasts quickly (weekly forecasts or so) you're not going to be able to do anything that is very statistically airtight, so that shouldn't be your focus. You should be able to detect obvious problems pretty quickly, though, and may be able to give the forecasters useful feedback.

In general, averaging the various forecasts is a pretty good strategy; you want to discard any seriously inferior forecasters first, though, which is where the stuff from the previous paragraph matters.