Statistical significance in binomial variable: how many tests needed? I have a problem that occurs randomly, in approximately 2 % of tests.
I am testing fixes, to prevent the error from occurring. 
The error occurring once would mean it is still happening and would render the solution invalid.
I would like to know, how many tests without the error occurring would give me a 95% confidence that the error has been fixed applying a certain solution? 
What is the equation of application here?
 A: We suppose that "95% confidence that the error has been fixed" means that there is (at most) a 5% probability that one finds no error amongst $n$ consecutive tests if the problem is still present. So, one has (at most) a 5% probability of concluding erroneously that the problem was solved if no error is observed in $n$ consecutive tests.
The probabilistic model that is useful in this set-up is the binomial distribution.
We choose to label the occurrence of an error in one test/experiment as a "success", which occurs with a probability $p$. In the question $p=0.02$.
The probability of observing $k$ "successes" in $n$ independent trials is
$$
P(X = k) =  \binom{n}{k} p^k (1-p)^{n-k}.
$$
Let $1-\alpha$ denote the required confidence level, so the probability of wrongly concluding that the problem is solved is $\alpha$. In the question $\alpha = 0.05$.
In order to reach the required level of confidence, we must have $P(X = 0) \leq \alpha$ if the problem is still present.
Solving this equation yields
$$
n \geq \frac{\log(\alpha)}{\log(1-p)} .
$$
So, with $p=0.02$ and $\alpha = 0.05$, the required number of trials is $149$.
To reach 99% confidence the required number of trials is $228$.
