1
$\begingroup$

I am not a statistics expert so please bear with me. It is about making predictions about how long something should take based on measurements of how long things took in the past. (Things like fixing software bugs)

Up until now I have been using a normal standard deviation to be able to state things like "the average bug will be fixed within 40 days, and 98% of bugs will be fixed within 115 days" since 115 is 3 standard deviations away from the mean. Then we can do things like establish SLAs (service level agreements) and promise that bugs will be fixed within 115 days or we pay a fee.

However I have been told that it is not appropriate to use standard deviation as lead times are not normally distributed, which makes sense, the normal distribution would enable us to say a small percentage of bugs will be fixed in -10 days for example. Lead times cannot be less than 0. I was told they might be Poisson distributed, which I am not sure about either, and that I could use Shewhart's method for calculating "the mean plus 2 sigma" and that would be a better value to use for lead times.

I haven't been able to find any information about this Shewhart's method. I did find something that said the standard deviation of a Poisson distribution was simply the square root of the sample size, but this doesn't make sense either, as I can imagine two different teams for example that both have 300 samples, and an average of say 40 days, but one team has 95% of lead times between 30 and 50, and none over 60, whereas the other team has 95% of lead times between 30 and 100, and some over 110. Simply using the square root of 300 will give the same standard deviation for both which does not seem to make sense.

Anyway I guess my questions are:

  1. What is Shewhart's method for working out something like "the mean plus 2 sigma", and
  2. If you don't know Shewhart's method, what would be a good formula to use to make statements like "x% of bugs are fixed within n days" based on measured lead time data. (Knowing that lead times cannot be less than 0, and are probably not normally distributed etc)

Thanks a lot, I am really out of my depth here.

$\endgroup$
  • $\begingroup$ This looks to me to be within the realm of what is known as survival analysis. It's a very broad field, but generally the normal distribution plays a fairly limited role. You also appear to have some natural covariates, like, e.g., which team a bug is assigned to. You probably also have some measure or notion of how difficult a bug will be to fix and how urgent it is. Ideally, you'd want to incorporate all of this into your analysis, but depending on your objectives, something simpler might suffice. $\endgroup$ – cardinal Jul 3 '11 at 16:33
  • $\begingroup$ Hi cardinal. I don't think any of these covariates are relevant in this case. I am interested in how a total system behaves. Also individual properties of individual bugs are not of interest either, just the mean lead time, and the mean plus 1, 2 or 3 sigma. $\endgroup$ – Kurt Jul 3 '11 at 20:14
  • $\begingroup$ Even when you are interested in a "total system", incorporating covariates could be very useful. Say, for example, you are drawing up a contract for some work and you already know which team it will be assigned to. Incorporating this information into your model can potentially protect your financial interest when drawing up the SLA. In fact, you seem to be providing such an example yourself, though maybe I've misunderstood the desired use of the model. $\endgroup$ – cardinal Jul 5 '11 at 0:00
2
$\begingroup$

Assuming you have an iid sample, you have a estimate of the distribution of lead times. Instead of using a normal approximation (what you were originally doing) or something similar to it, you could just used the empirical percentiles, which are consistent estimators of the true percentiles (by the law of large numbers), to make your claims. Specifically, if $X$ is the $p$-th percentile of your sample, then the claim that "$p$ proportion of bugs will be fixed within $X$ time" will be true with probability 1 as the sample size increases.

If you insist on using the poisson distribution, which specifies that if $X$ has a poisson distribution (say $X$ is your lead time in this case), then

$$ P(X=k) = \frac{\lambda^k e^{-\lambda}}{k!} $$

where $\lambda$ is a parameter estimated from the data (as the sample mean happens to be the maximum likelihood estimator). So your guess as the mass function would be

$$ P(X=k) = \frac{40^k e^{-40}}{k!}, $$

since you said the sample mean was 40. You'd then sum over this mass function to calculate quantities like the probability of a lead time being greater or less than a particular value. I don't see the need for some fancy method.

Edit: As noted in the comments below, if your lead times are discrete then something like the geometric distribution would be a much more natural parametric choice for their distribution than poisson.

$\endgroup$
  • $\begingroup$ Where does it say that the OP has an iid sample? $\endgroup$ – cardinal Jul 3 '11 at 16:29
  • $\begingroup$ nowhere I guess. I guess I assumed that because the normal approximation he was using (and his subsequent suggestions) seem to indicate this was a random sample from the same distribution. Also, I'd prefer to consider a tractable problem than an intractable one where there is arbitrary dependence within a non-normal sample :) I edited the first sentence to make it clear that I am the one assuming this. $\endgroup$ – Macro Jul 3 '11 at 16:32
  • $\begingroup$ I agree some simplifications and assumptions could take you a long way and, generally, preference should be given to simplicity where appropriate. I think you're on the right track regarding looking at empirical quantiles or something similar. Important covariates seem to be present, though. $\endgroup$ – cardinal Jul 3 '11 at 16:36
  • $\begingroup$ I'm not sure about the suggestion regarding the Poisson. (I know that wasn't your suggestion.) If we're going to model by a single parametric distribution for the marginal, then I doubt a Poisson is appropriate in a discrete setting. In a continuous setting, the knee-jerk reaction would be to use exponential waiting times, leading one to the geometric distribution for discrete times. (I'm not actually suggesting that as an approach, though.) $\endgroup$ – cardinal Jul 3 '11 at 16:38
  • 2
    $\begingroup$ Yes, that's correct (assuming the waiting times are iid). But the OP isn't asking about number of events, he's asking about a time until an event. $\endgroup$ – cardinal Jul 3 '11 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.