# How many must you sample with no negatives to conclude there is no negatives in the population?

If I have 1000 widgets made in my factory and I want to know that none of them are defective, how many must I sample to expect that there are no defects in the whole 1000 at 95% CL.

• Commented Dec 24, 2020 at 22:24
• Do you know the rates of false positives and false negatives for the process you use to test samples? Commented Dec 25, 2020 at 7:30
• Its safe to assume that there is not error on the test for the sample. Commented Dec 29, 2020 at 5:36
• In a comment to an answer you express scepticism about the simplicity of the answer (or maybe about its obviousness). For a rigorous demonstration, note that when $k$ of the widgets are defective, the chance that a sample of size $n$ contains none of them is $\binom{1000-k}{n}/\binom{1000}{n}.$ In the worst case where $k=1,$ then, your chance of not detecting it would be $$\binom{1000-1}{n}/\binom{1000}{n}=\frac{1000-n}{1000}.$$ The smallest $n$ that reduces this to $100-95=5\%$ or less is $950.$
– whuber
Commented Dec 30, 2020 at 14:43

As explained in the Wikipedia link, the "Rule of Three" is for (binomial) sampling with replacement or sampling from a theoetical infinite distribution. I am skeptical that this rule applies to your question.

Of course, to be 100% sure not even one of your 1000 widgets is defective, you will have to look at all of them.

If there is one defective widget among 1000, then your chances of finding it by looking at 750 randomly chosen widgets is only $$\frac{{1\choose 1}{999\choose 749}}{{1000\choose 750}} = 0.75,$$ a hypergeometric probability. In R,

dhyper(1,  1, 999,   750)
[1] 0.75


If you want to be 95% sure to find the one defective among 1000, you will need to sample 950 widgets (without replacement).

dhyper(1,  1, 999,   950)
[1] 0.95


If there are two or more defectives among the 1000, then it would be (almost) good enough to look at 750.

sum(dhyper(1:2,  2, 998,  750))
[1] 0.9376877
sum(dhyper(1:2,  2, 998,  775))
[1] 0.9495495
sum(dhyper(1:2,  2, 998,  777))
[1] 0.9504444

• In your calculation are you sure that you accounted for the ones you checked. If you look at 750 then you know those are good so you only need to check the 250 remaining. The 250 can act as the prediction set and the 750 is some sort of prior information. Perhaps there is a Bayesian formulation which makes this more clear. Commented Dec 24, 2020 at 22:34
• Hypergeometric probabilities are based on sampling without replacement, which is what one would do in your case. // A Bayesian approach might be a good idea. Commented Dec 24, 2020 at 22:37
• It is informative to think about trying to find a single defect in 1000 but a slightly different question. I will update if I make progress on the Bayesian approach. Commented Dec 24, 2020 at 23:06
• If you were to cheat and consider sampling with replacement, then the number of $k$ of defectives $n$ trials with defective probability $p$ is $\mathsf{Binom}(n,p).$ Then choose a noninformative Jeffreys prior $\mathsf{Beta}(.5,.5)$ on $p.$ If you have $k = 0$ successes in $950$ trials, Bayes Thm gives posterior $\mathsf{Beta}(.5,,950.5)$ and from R code qbeta(.95, .5, 950.5) gives Jeffreys posterior upper bound 0.002019247 on probability of defectives. // More properly for you, the issue is what convenient prior to use for hypergeometric data. Commented Dec 24, 2020 at 23:38
• However, it seems to me you may want to test a hypothesis of no defectives at the 5% level, given the minimal single defective among 1000 as alternative. That is the context of my Answer. // For CIs, it is easy to say 95% ("everyboty" uses that), but not so easy to say what upper bound you'd want to have for a one-sided CI. Commented Dec 24, 2020 at 23:45

Suppose we want to expect that there are no defective widgets at the 95% confidence level. What this means is that we must test enough widgets that if there were a defective widget, then we would have a 95% chance of finding the defective one in our tests.

Of course, if we want to have a 95% chance of finding the defective widget, that means we must test 95% of all widgets. So the answer, unfortunately, is that you have to test 950 of the 1,000 widgets.

And that's it, that's the entire answer to your question. But let me talk about practical considerations, because in practice, there isn't really any situation in which you'd want to test 95% of all widgets.

Scenario 1: You're building a chain out of 1,000 links. For some reason, you've decided to buy each link from a different person, so you have 1,000 different people supplying chain links to you. All of the links have to be strong in order for the chain to be strong; having even one weak link is unacceptable.

In this scenario, the problem is that testing some of the links doesn't tell you anything about the remaining links. Even if you test 999 of them, you still haven't learned anything whatsoever about the remaining link. So, testing (say) only 750 of the links is definitely not enough.

As I mentioned, in order to conclude at the 95% confidence level that none of the links are weak, you need to test 950 of them. At this point, you might be wondering, "Why should I stop at 950? Why not just test all 1,000 of them?" And the answer is that you're absolutely right. You should probably just test all 1,000 links, so that you know that all of them are strong.

Scenario 2: You have a machine that makes chain links, and you've just made a batch of 1,000 chain links using that machine. As above, all of the links have to be strong in order for the chain to be strong. How many chain links should you test in order to be sure that you'll have a strong chain?

If you're faced with this scenario in the real world, then you should try to find out more information about the machine. The best case is that there are two types of machines: ones which produce only strong links, and ones which produce only weak links. In this case, you only need to test one link in order to know that all of the links are strong!

A more realistic case, perhaps, is where some machines produce only strong links, and other machines produce 50% strong links and 50% weak links. In this case, in order to achieve your 95% confidence level, you only need to test 5 links and see that they're all strong.

Another interesting case is the case where these machines are known to be very reliable, and 99.9% of them produce only strong links. In this case, if this machine was chosen randomly out of all machines, then you don't need to test any links in order to reach the 95% confidence level that all of the links are strong.

I can't describe all possible practical situations, of course, but hopefully this gives you an idea of why 950 is the correct answer to the original question, as well as why that answer isn't likely to be very useful in practice.

• This seems too simple to be correct but I do not have a good argument against this way of thinking. Commented Dec 29, 2020 at 5:44
• @Keith Sometimes, the answer is simple. This is one of those times. That said, there are some practical considerations that I haven't talked about in my answer. I think I'll edit my answer to talk about those. Commented Dec 30, 2020 at 1:30
• The connection this made for me was that 95% confidence of 100% being good is the same as 100% confidence of 95% being good. Commented Jan 4, 2021 at 18:37

I looked at a version of this problem back in April, to see what we could say about Covid elimination based on reasonable numbers of tests (spoiler: not a lot).

As @keith's answer shows, if you want to be sure there isn't even one failure in the population, you need sample a large fraction of the population. That's obviously not sensible in most cases.

As @BruceET says in his comment, a Bayesian solution makes sense. However, because there's very little information in the data about very small numbers of failures in the population

• it will matter what prior you put on very small numbers of failures
• if you think small numbers are plausible a priori, and don't sample a big fraction of the population, you will inevitably still think that a posteriori, so you won't end up with high posterior probability on zero

Suppose you take a $$Beta(a,b)$$ prior for the probability of failure. The posterior after no failures out of $$n$$ is $$Beta(a+0,b+n)$$. So you can look up the quantiles of that distribution and see when it's concentrated close enough to zero for what you want to use it for.

• This is useful. Can you add more. Perhaps an example. We know the probably of defect is small (<1%) and would want to not assume much more than that. I was thinking to take the only prior information to be the sample but could not sort out the formulation. Commented Dec 24, 2020 at 23:11

The Rule of three states that if I sample N then I know I have a rate less than 3/N at 95% CL

My expectation, E, is my rate times the number of total events, T.

E = T*3/N

The total events remaining is just the number unsampled T=1000-N

E = (1000-N)*3/N

I want to expect at most 1 so E=1

Solving for N gives

N = 10003/(1+3) = 10003/4 = 750

Sampling 3/4 seems like a lot so I have some doubts but this is a pretty strict requirement.

The general case for any confidence can also be given where the multiplayer is -ln(α) with α being 1 minus the confidence level.

With a population size P the general case is

N = P*ln(α)/(ln(α)-E)

• I think you've calculated the wrong thing. By setting $E = 1$, you've calculated the number of widgets that would need to be sampled in order to expect that no more than 1 widget is defective. But we want to ensure that there are 0 defective widgets, not no-more-than-1 defective widget. Commented Dec 25, 2020 at 4:29
• I do realize that. I think that this works for my situation but you are rigorously correct. I thought about putting it to 0.49 then I could round down to 0. I am unclear how to handle the discrete nature of this problem. Commented Dec 29, 2020 at 5:41