Doing some code testing and I have a pre-fix environment where I ran 24 test scenarios and 20 out of 24 worked as expected—only 4 (16.67%) failed.

In the code where the fix exists, if I do another 24 tests, what are the chances that I get a 100% pass rate even if the code didn't fix the problem?

I feel like I probably need to do more tests but I can't remember my stats course and was hoping someone here could help

  • $\begingroup$ Is it a homework? $\endgroup$ – Tim Aug 21 at 16:22
  • 1
    $\begingroup$ No, actual work work—I'm doing QA testing for code & trying to figure out just how hard I have to test this to be confident the code fixes the problem $\endgroup$ – Ponch22 Aug 21 at 17:11
  • $\begingroup$ If these are 24 different scenarios, then you don't have nearly enough data to even start an analysis. If they're the same, we can sort of guess the values. Can you clarify? $\endgroup$ – Mooing Duck Aug 22 at 0:36
  • $\begingroup$ "What are the chances that I get a 100% pass rate even if the code didn't fix the problem?" Unless your tests are not deterministic, there's a 0% chance of this happening. I don't think using probability to approach this is the right idea; you should formalize the bug you're trying to fix, and create tests directly relating to it. You could write 100 more tests that will pass post-fix, but without analysing the tests themselves, that result is meaningless, as they might be testing the wrong thing. $\endgroup$ – Rob Aug 22 at 1:30
  • $\begingroup$ To expand on what some other comments indicated: Statistics, fundamentally, argues about probabilities surrounding random events. Why do you believe that any of these events are random? $\endgroup$ – DreamConspiracy Aug 22 at 7:52

One straightforward way to analyze this situation is to assume for testing purposes that the fix made no difference. Under this assumption, you may view the assignment of the (potential) 48 observations into pre-fix and post-fix groups as being random. If (hypothetically) all the post-fix outcomes work as expected, it means you have observed 44 expected outcomes and 4 "failures" out of 48 (presumably) independent observations.

This permutation test frames your question as "what is the chance that a random partition of 48 test scenarios, of which four are failures, places all four failures in the first group?" Because all such random partitions are equally probable, this reduces the answer to a simple counting calculation:

  • There are $24\cdot 23 \cdot 22 \cdot 21$ equally-likely ways to place the four failures within the sequence of $24$ pre-fix scenarios.

  • There are $48\cdot 47\cdot 46 \cdot 45$ equally-likely ways to place the four failures within the sequence of all $48$ scenarios.

Ergo, under these hypotheses the chances that all four failures occurred before the fix are

$$\frac{24\cdot 23 \cdot 22 \cdot 21} {48\cdot 47\cdot 46 \cdot 45} = \frac{77}{1410} \approx 5.5\%.$$

The interpretation could be written in the following way:

Suppose four of 24 initial scenarios resulted in failure and none of 24 post-fix scenarios resulted in failure. This had a $5.5\%$ chance of occurring when the fix changed nothing. Because $5.5\%$ is small, it provides some evidence that the fix worked; but because it's not really that small--events with those chances happen all the time--it's probably not enough evidence to convince anyone who is sceptical that the fix worked. Unless collecting more data is very expensive, it might not be enough evidence to support consequential decisions related to the fix.

Even if the fix has already occurred, you have the flexibility to collect more post-fix data. You can experiment with analogous computations to see how many more post-fix scenarios you need to run (all without failure) in order to establish a convincing level of evidence that something changed for the better after the fix. For instance, if you need this chance to be less than $1\%,$ you would want to run $49$ post-fix scenarios rather than $24.$

Of course, as soon as one post-fix scenario produces a failure you would likely stop your testing and try to improve the fix. Sophisticated versions of this approach (where you stop once you have enough information, without trying to guess how much you need beforehand) are called sequential testing procedures.

  • $\begingroup$ I like this... And to make sure I understand, running 36 post-fix scenarios I'd have a ~97.8% confidence all four failures weren't just in the pre-fix scenarios by chance! $\endgroup$ – Ponch22 Aug 21 at 19:12
  • $\begingroup$ Yes, that's the number I get. $\endgroup$ – whuber Aug 21 at 19:24
  • $\begingroup$ I might have to do more pre-fix tests, so let me see if I TRULY understand the math... Let's say I do another 8 pre-fix tests and see 3 new errors. Now with N = 32 I have 7 errors or a 21.9% fail rate If I then do 24 additional post-fix tests and see 0 errors in those 24, I'd have ~98.5% confidence the fix worked because there would only be a 1.45% chance the 7 errors were localized within the first 32 by chance: (32⋅31⋅30⋅29⋅28⋅27⋅26) / (56⋅55⋅54⋅53⋅52⋅51⋅50) ≈ 1.45% (It's sad I was a math major in college, but I always had trouble with stats!) $\endgroup$ – Ponch22 Aug 21 at 20:47
  • $\begingroup$ That's right, you got the pattern. $\endgroup$ – whuber Aug 21 at 21:29
  • 2
    $\begingroup$ @Ponch While I think whuber's answer to the posted question is pitched perfectly, your extended questions in followup suggest that it may be worth noting that the relevant distribution here is called the hypergeometric distribution, and with which many programs can do the calculations for you, perhaps making it a little easier to figure out whatever you're trying to work out. It may save you some effort. e.g. Excel has a function for it. $\endgroup$ – Glen_b -Reinstate Monica Aug 21 at 23:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.