# Statistical significance in binomial variable: how many tests needed?

I have a problem that occurs randomly, in approximately 2 % of tests. I am testing fixes, to prevent the error from occurring. The error occurring once would mean it is still happening and would render the solution invalid.

I would like to know, how many tests without the error occurring would give me a 95% confidence that the error has been fixed applying a certain solution?

What is the equation of application here?

We suppose that "95% confidence that the error has been fixed" means that there is (at most) a 5% probability that one finds no error amongst $n$ consecutive tests if the problem is still present. So, one has (at most) a 5% probability of concluding erroneously that the problem was solved if no error is observed in $n$ consecutive tests.
The probabilistic model that is useful in this set-up is the binomial distribution. We choose to label the occurrence of an error in one test/experiment as a "success", which occurs with a probability $p$. In the question $p=0.02$.
The probability of observing $k$ "successes" in $n$ independent trials is $$P(X = k) = \binom{n}{k} p^k (1-p)^{n-k}.$$
Let $1-\alpha$ denote the required confidence level, so the probability of wrongly concluding that the problem is solved is $\alpha$. In the question $\alpha = 0.05$.
In order to reach the required level of confidence, we must have $P(X = 0) \leq \alpha$ if the problem is still present. Solving this equation yields $$n \geq \frac{\log(\alpha)}{\log(1-p)} .$$ So, with $p=0.02$ and $\alpha = 0.05$, the required number of trials is $149$. To reach 99% confidence the required number of trials is $228$.
• Perfect answer! Exactly what I needed and greatly explained. For what it's worth, I had reached to get the binom.test function in R which gives you values for confidence intervals too. But it was lovely to get the whole explanation. Thanks. – xmar Jul 21 '14 at 10:58