I am not a statistics expert so please bear with me. It is about making predictions about how long something should take based on measurements of how long things took in the past. (Things like fixing software bugs)
Up until now I have been using a normal standard deviation to be able to state things like "the average bug will be fixed within 40 days, and 98% of bugs will be fixed within 115 days" since 115 is 3 standard deviations away from the mean. Then we can do things like establish SLAs (service level agreements) and promise that bugs will be fixed within 115 days or we pay a fee.
However I have been told that it is not appropriate to use standard deviation as lead times are not normally distributed, which makes sense, the normal distribution would enable us to say a small percentage of bugs will be fixed in -10 days for example. Lead times cannot be less than 0. I was told they might be Poisson distributed, which I am not sure about either, and that I could use Shewhart's method for calculating "the mean plus 2 sigma" and that would be a better value to use for lead times.
I haven't been able to find any information about this Shewhart's method. I did find something that said the standard deviation of a Poisson distribution was simply the square root of the sample size, but this doesn't make sense either, as I can imagine two different teams for example that both have 300 samples, and an average of say 40 days, but one team has 95% of lead times between 30 and 50, and none over 60, whereas the other team has 95% of lead times between 30 and 100, and some over 110. Simply using the square root of 300 will give the same standard deviation for both which does not seem to make sense.
Anyway I guess my questions are:
- What is Shewhart's method for working out something like "the mean plus 2 sigma", and
- If you don't know Shewhart's method, what would be a good formula to use to make statements like "x% of bugs are fixed within n days" based on measured lead time data. (Knowing that lead times cannot be less than 0, and are probably not normally distributed etc)
Thanks a lot, I am really out of my depth here.