If I have an issue that causes a failure at a rate of, say, 1 in 20 cycles and then deploy a fix, how do I work out how many test cycles to perform to be confident to 95% that the fix worked?

I know this is possibly straightforward but I can't find the right thing when searching!

Edit: Trying to re-phrase and add more detail.

Assume that the fix is supposed to be a full fix (i.e. stop the issue occurring again). I am trying to work out how many cycles to perform without a failure to be, say, 95% certain that the issue is fixed and not just that it is not seen as I did not run enough cycles.

  • $\begingroup$ First of all we need to define "fixed", is fixed a 1/50 rate for example? 1/100? In any case, a proper answer would probably employ Zhu (2016) Sample Size Calculation for Comparing Two Poisson or Negative Binomial Rates in Noninferiority or Equivalence Trials. That said, if we are willing to assume that we care for a difference in binomial proportions, we can do a standard proportions power calculations like: power.prop.test( p1= 1/20, p2 = 1/50, power = 0.90, alternative = "one"); in that sense about 650 samples suffice. $\endgroup$
    – usεr11852
    May 17, 2019 at 11:45
  • $\begingroup$ The failure rate is 1/20 before fix. If the failure rate is 1/10 after fix, your confidence will decrease as the number of test cycles increases. So your question has no answer, because you did not provide enough information. $\endgroup$
    – user158565
    May 18, 2019 at 3:31
  • $\begingroup$ @user158565: 1/20 < 1/10, so if anything the fix is detrimental in that case. (But yeah, we need a bit more info.) :) $\endgroup$
    – usεr11852
    May 23, 2019 at 8:10

1 Answer 1


Provided your test cycles are independent trials, this looks like a typical case for a binomial confidence interval.

There is no minimum number of test cycles for a 95% confidence interval (other than that it has to be positive), but the more you do the smaller your confidence interval will be. If you use a large number of trials you can use the normal approximation.

$CI=\hat{p} \pm z \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$

where $\hat{p}$ is the estimated failure rate, and $z$ is the number corresponding to a 95% range of the standard normal distribution (i.e. 1.96), and $n$ is the number of test cycles. As you can see, $n$ only serves to shrink your interval size, and of course it is part of your estimate for $\hat{p}$.

The normal approximation can be used for large sample sizes, Wikipedia offers several other possibilities.

  • $\begingroup$ I think this is what I am after. Let me give more background: I have previously seen, in a software issue tracking system, where you log how often an issue occurred (e.g. 1 in 20 tests) and then the system gives the number of cycles to use in validation to get confidence to 90%, 95%, 99%. I am trying to figure out how that is calculated. Is that what your answer does (re-arranging for n, but then what is CI)? $\endgroup$
    – SimpleOne
    May 17, 2019 at 11:56
  • $\begingroup$ CI are the boundaries for the confidence interval (note the $\pm$ in the formula, so it's really two values). Given a certain absolute width around $\hat{p}$, one could set the substraction of those two boundaries equal to that absolute width, rearrange and solve for $n$. For example, if I want my 95% interval to be 2 percent points wide, I'd set the difference in the boundaries to 0.02, and then solve. Note that $z$ varies for different confidence intervals (it's only 1.96 for the 95%, for 90% it's 1.65) $\endgroup$ May 17, 2019 at 12:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.