Error Rate Confidence

Question

I have a very basic understanding of statistics and have the following question:

Given a test that can be run and give either a Pass or Fail result, how do I calculate how many times I must run the test to give a certain confidence interval for the Fail rate?

Specifically, if I think the Fail rate is low, like lower than 1 / 1,000,000,000, how many times do I need to run the test, presumably getting Pass every time to ensure the Fail rate is at most 1 / 1,000,000 with 99% confidence?

Any keywords/terms to search on are welcome.

Answer 1

Look at binomial confidence intervals (specifically one-sided, since you don't have a lower bound)

$$CI = p \pm z_{1-\alpha/2}\sqrt\frac{p(1-p)}{n}$$

$z$ is the z-value for a certain level of confidence (2.326 for 99% confident). Change this depending on how confident you want to be that $p$ is within the interval.

$p$ is the estimated probability.

$n$ is the number of trials.

For example, if you ran 10000 trials and 2 failed, your $p$ would be $.0002$ and your $n$ would be $10000$.

Answer 2

Please do not use the normal CI! The Clopper-Pearson CI is much better for extremely low fail rates. For app. 95% confidence, this can be simplified to 3/N (Rule of Three). This means if you e.g. run 100 tests, and get all the time a pass. Then you cannot guarantee pfail=0, but approx. pfail<3/100=3%. This formula works for NO fails, but for any count N. For higher confidence, like 99%, the number "3" would be a bit larger. If you have fails, then better go for the Clopper-Pearson formula.