Determine the sample size to validate a bug fix

Question

Using gpower, I would like to calculate the sample size to validate a bug fix.

There is a software bug that appears in 3 out of 1000 test runs. I would like to validate a bug fix so that it doesn't appear within a degree of confidence.

My question is what kind of "test family" do I choose for that? I am a bit overwhelmed by the number of choices. Also, when trying the different test families I seem to never enter the failure rate (0.003) which is crucial to make the prediction right?

I can enter the following parameters

• power of 0.80
• alfa / err prob of 0.05
• effect size of 0.2

For fun, I also asked chatgpt for an answer and it gave back a sample size that does take into account the failure rate of 0.003

n = (Zbeta + Zalpha)^2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p1 - p2)^2

Plugging in the values, we get:

n = (0.84 + 1.96)^2 * ((0.003 * (1 - 0.003)) + (0 * (1 - 0))) / (0.003 - 0)^2
n = 1472.39

Thanks for setting up this beginner! I would like to master gpower a bit better for my daily work. If gpower isn't suited for this at all that would be an answer as well :)

Update 20230416

I think I can now zoom in on a better question. Given that a test fails 3 out of 1000, what is the chance it fails 0 out of 1000? And then, what is the chance it fails 99% of the time 0 out of 1000?

So the question is more: How do I prove that the previous failure rate has gone? Rather then to prove my bugifx is perfect since you can't prove perfection in a limited time.

Answer 1

Saying that a bug is "fixed" might be taken to mean that it has zero probability of occurring. That's not something you can determine. You might still have a very low probability, less than your current 0.003, that would only be seen in a very large sample.

For this situation you might apply the "rule of three". If there are no occurrences of a rare event in $N$ trials, then you have a 95% confidence interval for the true probability within the range $[0,3/N]$.* As the linked Wikipedia page shows, with the large number of trials you need to use, that approximation is indistinguishable from the "exact binomial" analysis that's needed with smaller samples, where design might be improved with software like G*Power.

So if you did 1 million trials and found no failures, you would have a 95% confidence interval from 0 to 3 out of 1 million. The Wikipedia page shows how to adjust the value of 3 to obtain other confidence intervals. For example, the 99.5% confidence interval following such a test would be between 0 and 5.3 out of 1 million.

The above assumes that the trials are independent and have identical probabilities of failure. That might not always hold in tests of software bugs, so be warned.

In response to comments:

First, in a comment on the question, @dipetkov rightly notes that this situation should only occur with non-deterministic code, with a link to better ways to deal with this situation in testing computer code.

Second, from the perspective of ruling out rare independent events in general, there is no "shortcut" to extended testing.

If the probability of an event in a single trial is $p$, then the probability of no events in $N$ independent trials is $(1-p)^N$. With your scenario of $p=0.003$ and $N=1000$, the probability of observing 0 events is 0.0496. There is about a 1% chance, after 1000 trials, of observing no events with $p$ as high as 0.0046.

You can use that formula to evaluate your tradeoffs among assumed probabilities, the risk of missing an event given that probability, and the number of trials. But that formula is equivalent to what the "rule of three" (and its extensions to other confidence intervals) provides in the limit of large $N$.

---

*Note that the sample size of 1472 that you show in the question only provides a 95% confidence interval of [0, 0.002] if there are no failures.