I have a very basic understanding of statistics and have the following question:

Given a test that can be run and give either a Pass or Fail result, how do I calculate how many times I must run the test to give a certain confidence interval for the Fail rate?

Specifically, if I think the Fail rate is low, like lower than 1 / 1,000,000,000, how many times do I need to run the test, presumably getting Pass every time to ensure the Fail rate is at most 1 / 1,000,000 with 99% confidence?

Any keywords/terms to search on are welcome.


2 Answers 2


Look at binomial confidence intervals (specifically one-sided, since you don't have a lower bound)

$$CI = p \pm z_{1-\alpha/2}\sqrt\frac{p(1-p)}{n}$$

$z$ is the z-value for a certain level of confidence (2.326 for 99% confident). Change this depending on how confident you want to be that $p$ is within the interval.

$p$ is the estimated probability.

$n$ is the number of trials.

For example, if you ran 10000 trials and 2 failed, your $p$ would be $.0002$ and your $n$ would be $10000$.


Please do not use the normal CI! The Clopper-Pearson CI is much better for extremely low fail rates. For app. 95% confidence, this can be simplified to 3/N (Rule of Three). This means if you e.g. run 100 tests, and get all the time a pass. Then you cannot guarantee pfail=0, but approx. pfail<3/100=3%. This formula works for NO fails, but for any count N. For higher confidence, like 99%, the number "3" would be a bit larger. If you have fails, then better go for the Clopper-Pearson formula.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.