1
$\begingroup$

I have a problem that occurs randomly, in approximately 2 % of tests. I am testing fixes, to prevent the error from occurring. The error occurring once would mean it is still happening and would render the solution invalid.

I would like to know, how many tests without the error occurring would give me a 95% confidence that the error has been fixed applying a certain solution?

What is the equation of application here?

$\endgroup$
2
$\begingroup$

We suppose that "95% confidence that the error has been fixed" means that there is (at most) a 5% probability that one finds no error amongst $n$ consecutive tests if the problem is still present. So, one has (at most) a 5% probability of concluding erroneously that the problem was solved if no error is observed in $n$ consecutive tests.

The probabilistic model that is useful in this set-up is the binomial distribution. We choose to label the occurrence of an error in one test/experiment as a "success", which occurs with a probability $p$. In the question $p=0.02$.

The probability of observing $k$ "successes" in $n$ independent trials is $$ P(X = k) = \binom{n}{k} p^k (1-p)^{n-k}. $$

Let $1-\alpha$ denote the required confidence level, so the probability of wrongly concluding that the problem is solved is $\alpha$. In the question $\alpha = 0.05$.

In order to reach the required level of confidence, we must have $P(X = 0) \leq \alpha$ if the problem is still present. Solving this equation yields $$ n \geq \frac{\log(\alpha)}{\log(1-p)} . $$ So, with $p=0.02$ and $\alpha = 0.05$, the required number of trials is $149$. To reach 99% confidence the required number of trials is $228$.

$\endgroup$
1
  • $\begingroup$ Perfect answer! Exactly what I needed and greatly explained. For what it's worth, I had reached to get the binom.test function in R which gives you values for confidence intervals too. But it was lovely to get the whole explanation. Thanks. $\endgroup$ – xmar Jul 21 '14 at 10:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.