# Number of tests to perform to get confidence rate

If I have an issue that causes a failure at a rate of, say, 1 in 20 cycles and then deploy a fix, how do I work out how many test cycles to perform to be confident to 95% that the fix worked?

I know this is possibly straightforward but I can't find the right thing when searching!

Edit: Trying to re-phrase and add more detail.

Assume that the fix is supposed to be a full fix (i.e. stop the issue occurring again). I am trying to work out how many cycles to perform without a failure to be, say, 95% certain that the issue is fixed and not just that it is not seen as I did not run enough cycles.

• First of all we need to define "fixed", is fixed a 1/50 rate for example? 1/100? In any case, a proper answer would probably employ Zhu (2016) Sample Size Calculation for Comparing Two Poisson or Negative Binomial Rates in Noninferiority or Equivalence Trials. That said, if we are willing to assume that we care for a difference in binomial proportions, we can do a standard proportions power calculations like: power.prop.test( p1= 1/20, p2 = 1/50, power = 0.90, alternative = "one"); in that sense about 650 samples suffice. May 17, 2019 at 11:45
• The failure rate is 1/20 before fix. If the failure rate is 1/10 after fix, your confidence will decrease as the number of test cycles increases. So your question has no answer, because you did not provide enough information. May 18, 2019 at 3:31
• @user158565: 1/20 < 1/10, so if anything the fix is detrimental in that case. (But yeah, we need a bit more info.) :) May 23, 2019 at 8:10

$$CI=\hat{p} \pm z \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$$
where $$\hat{p}$$ is the estimated failure rate, and $$z$$ is the number corresponding to a 95% range of the standard normal distribution (i.e. 1.96), and $$n$$ is the number of test cycles. As you can see, $$n$$ only serves to shrink your interval size, and of course it is part of your estimate for $$\hat{p}$$.
• CI are the boundaries for the confidence interval (note the $\pm$ in the formula, so it's really two values). Given a certain absolute width around $\hat{p}$, one could set the substraction of those two boundaries equal to that absolute width, rearrange and solve for $n$. For example, if I want my 95% interval to be 2 percent points wide, I'd set the difference in the boundaries to 0.02, and then solve. Note that $z$ varies for different confidence intervals (it's only 1.96 for the 95%, for 90% it's 1.65) May 17, 2019 at 12:49