# Combining expert estimates

I am trying to work on a process to improve how how well my team estimates. I want to look at using some statistics to help out and embrace the uncertainty in how we estimate tasks.

If I have a group of experts and I ask them for a best (b), average (a) and worst (w) case estimate for a task, creating a triangular distribution.

Using a monte carlo method I can generate 10,000 trials using the triangular distribution for each of the estimators and combine these together into one sample set.

I can then use a MLE to create a beta distribution from the sample statistics, then use the cdf(x) functions to determine the likelihood of an estimated number of days, or use the inverseF to determine the number of days to maintain a certain likelihood.

This seems to make sense to me logically, however I am unsure about the fitting of a Beta distribution to the samples. I am wondering if there is a better distribution I should be using, especially given that the samples are just randomly generated from N triangular distributions.

Does anybody have any suggestions or experience in doing this that could point me in the right direction? It's been a few years since I left uni, so my statistics is a little rusty. There are plenty of examples of this process for one expert, however I would like to be able to do the same thing for multiple experts.

Thank you,

Aidan

-

I think you're being a little over-ambitious --- using the beta distribution vs. another distribution function is the least of your problems. I'm not really sure what resampling the forecasts is supposed to achieve, and then re-estimating some parameters based on those resampled data isn't getting you anything that couldn't be estimated from the original forecasts.

A better approach might be to track the estimate and the actual outcome over time and keep track of some other information about the task. Then you can look at bias (i.e. the average difference between the estimate and the outcome) and how the bias depends on other task characteristics. Unless you're in a setting where you have a lot of historical data (offhand, I'd say estimates and outcomes for ~ 20 past tasks if you don't car about conditioning on characteristics; more if you do, but that number's just off the top of my head) or are going to be accumulating new forecasts quickly (weekly forecasts or so) you're not going to be able to do anything that is very statistically airtight, so that shouldn't be your focus. You should be able to detect obvious problems pretty quickly, though, and may be able to give the forecasters useful feedback.

In general, averaging the various forecasts is a pretty good strategy; you want to discard any seriously inferior forecasters first, though, which is where the stuff from the previous paragraph matters.

-
I completely agree with you about historical data, but there is a boostrapping problem that I can't overcome - the fact that we have no historical data. I was using the monte-carlo as a mechanism of artificially creating the historical data. By basically running the project 10,000 times using a random value between the prescribed values I can get an indication of the likely estimates. –  Aidos Dec 13 '11 at 8:12
I don't think that artificially creating historical data is going to help; you're still basing everything on the few real observations that you've recorded. Until you accumulate enough data, you can get a feel for the uncertainty among your experts by looking at the a measure of the spread of their forecasts -- the standard deviation or the median absolute difference. –  Gray Dec 13 '11 at 8:54
I can understand your point, this technique is not something I have invented - it's a quite commonly used approach for measuring variance and uncertainty in estimates. My problem is that all of the examples I can find deal with one source of estimates, whereas I want to be able to work with multiple sources of estimates and combine them. –  Aidos Dec 13 '11 at 10:12
I realize that you didn't invent the bootstrap, but what you're describing isn't it. For a simplistic bootstrap, you'd start with data and an estimator; resample n observations (n is the sample size) a bunch of times with replacement; calculate the estimator on each resample; then use the distribution of the statistic across draws to characterize its uncertainty. But it's an asymptotic technique --- if you have 10 observations, it's going to do badly. The MLE and beta distribution part seems to be your own invention, but I'd be happy to look at a reference. –  Gray Dec 13 '11 at 14:03