Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I am trying to work on a process to improve how how well my team estimates. I want to look at using some statistics to help out and embrace the uncertainty in how we estimate tasks.

If I have a group of experts and I ask them for a best (b), average (a) and worst (w) case estimate for a task, creating a triangular distribution.

Using a monte carlo method I can generate 10,000 trials using the triangular distribution for each of the estimators and combine these together into one sample set.

I can then use a MLE to create a beta distribution from the sample statistics, then use the cdf(x) functions to determine the likelihood of an estimated number of days, or use the inverseF to determine the number of days to maintain a certain likelihood.

This seems to make sense to me logically, however I am unsure about the fitting of a Beta distribution to the samples. I am wondering if there is a better distribution I should be using, especially given that the samples are just randomly generated from N triangular distributions.

Does anybody have any suggestions or experience in doing this that could point me in the right direction? It's been a few years since I left uni, so my statistics is a little rusty. There are plenty of examples of this process for one expert, however I would like to be able to do the same thing for multiple experts.

Thank you,

Aidan

share|improve this question

2 Answers 2

I think you're being a little over-ambitious --- using the beta distribution vs. another distribution function is the least of your problems. I'm not really sure what resampling the forecasts is supposed to achieve, and then re-estimating some parameters based on those resampled data isn't getting you anything that couldn't be estimated from the original forecasts.

A better approach might be to track the estimate and the actual outcome over time and keep track of some other information about the task. Then you can look at bias (i.e. the average difference between the estimate and the outcome) and how the bias depends on other task characteristics. Unless you're in a setting where you have a lot of historical data (offhand, I'd say estimates and outcomes for ~ 20 past tasks if you don't car about conditioning on characteristics; more if you do, but that number's just off the top of my head) or are going to be accumulating new forecasts quickly (weekly forecasts or so) you're not going to be able to do anything that is very statistically airtight, so that shouldn't be your focus. You should be able to detect obvious problems pretty quickly, though, and may be able to give the forecasters useful feedback.

In general, averaging the various forecasts is a pretty good strategy; you want to discard any seriously inferior forecasters first, though, which is where the stuff from the previous paragraph matters.

share|improve this answer
    
I completely agree with you about historical data, but there is a boostrapping problem that I can't overcome - the fact that we have no historical data. I was using the monte-carlo as a mechanism of artificially creating the historical data. By basically running the project 10,000 times using a random value between the prescribed values I can get an indication of the likely estimates. –  Aidos Dec 13 '11 at 8:12
    
I don't think that artificially creating historical data is going to help; you're still basing everything on the few real observations that you've recorded. Until you accumulate enough data, you can get a feel for the uncertainty among your experts by looking at the a measure of the spread of their forecasts -- the standard deviation or the median absolute difference. –  Gray Dec 13 '11 at 8:54
    
I can understand your point, this technique is not something I have invented - it's a quite commonly used approach for measuring variance and uncertainty in estimates. My problem is that all of the examples I can find deal with one source of estimates, whereas I want to be able to work with multiple sources of estimates and combine them. –  Aidos Dec 13 '11 at 10:12
    
I realize that you didn't invent the bootstrap, but what you're describing isn't it. For a simplistic bootstrap, you'd start with data and an estimator; resample n observations (n is the sample size) a bunch of times with replacement; calculate the estimator on each resample; then use the distribution of the statistic across draws to characterize its uncertainty. But it's an asymptotic technique --- if you have 10 observations, it's going to do badly. The MLE and beta distribution part seems to be your own invention, but I'd be happy to look at a reference. –  Gray Dec 13 '11 at 14:03

I don't know how to answer your question. However, I want to give one slight word of caution about trusting the "worst case" and "best case" estimates too much.

There has been a lot of research suggesting that people tend to be overly confident in their estimates, and to underestimate the degree of uncertainty in their answers. For instance, you can give people a 20-question quiz, where each quiz question asks them something they don't know (e.g., "How many gas stations are there in the US?"), but rather than asking them to give the answer, you ask them to give a 90% confidence interval. Then, you grade their quiz by marking their answer correct if the true answer falls within their chosen interval, and false otherwise. If we were perfect at estimating our own level of confidence accurately, you would expect that people would score on average about 18 out of 20 questions correct. Instead, what you find is that most people's scores are much lower. In other words, they choose a confidence interval that is too narrow, underestimating their lack of knowledge or the amount of uncertainty in their estimate.

Therefore, if you ask people for the best case and worst case time to complete a task, I would expect their worst-case estimates to be too optimistic: I would expect that the true worst case may be significantly worse than their estimates would indicate. This bias seems to be a fundamental human phenomenom, that is difficult to counter (even for experts). So, just beware that the amount of uncertainty in your estimates may be greater than your polls of the experts may indicate. I don't know of any easy way to correct for this through statistics; it is just something you'll have to be aware of and keep in mind.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.