I regularly inspect finite populations for failures (we make custom products in batches of ~500-800). Currently, we inspect every product for failure, which is quite a bit of work. I want to reduce the number of samples we inspect by stating a desired error rate, and determining the number of samples needed to inspect to be confident we are hitting the error rate.

I'm aware that the the rule of 3 would apply if there are no failures, but I'd like a more precise solution in situations where the sampling does have a failure.

This seems like it would be best modeled as a hypergeometric distribution, but I'm struggling to frame the question properly in these terms. I like examples, so say I have a population of 500, and I want to be 99% confident that there are 5 or fewer failures in the population.

How do I frame this type of question using the hypergeometric distribution?

My current attempt is this (in terms of the Wiki variables):

$N=500$; $K=495$; $n=100$; $P(X)=0.01$

With a sample of 50 and 3 failures, the probability of a 10% failure rate is (for some reason, I'm getting a [Math Processing Error] when using LaTex, so I'll post my progress in R commands)

qhyper(p=0.01, m=495, n=5, k=100)

Which gives $k=97$. Interpreting this, I should take away that from a 100 sample set in a population of 500, I can be 99% confident that if I find 3 or fewer failures in my sample, the error rate is no worse than 1% (or 5 in 500)?

I admittedly don't have a great intuition for this type of distribution, but my gut is giving me pause at the thought of sampling 100 specimens, finding 3 failures, and declaring with 99% confidence that there are only up to 2 more in the remaining 400.

  • 1
    $\begingroup$ Rule of 3 is actually not the best choice: stats.stackexchange.com/questions/134380/… $\endgroup$
    – Tim
    Jan 29, 2016 at 21:01
  • $\begingroup$ Thank you for the update, @Tim. I'll certainly use a different approach when there are no failures. I'm still interested, though, in a solution when failures do exist. $\endgroup$
    – Ashe
    Feb 3, 2016 at 16:24

1 Answer 1


Suppose you sample $n$ members from a population of $N$ (without replacement) and $k$ of them are failures. The definition of confidence tells us to ask this question:

If there are $K$ failures in the population, what is the chance we observe $k$ or fewer failures in the sample?

Without going into the combinatorial details, let's just call this number $p(k,K;n,N)$. It can be used to establish upper confidence limits on $K$ through a form of logical inversion. Let $N$ and $n$ both be known and $\alpha$ be a specified probability. If $K^\prime$ is so large that $p(k,K^\prime;n,N) \lt \alpha$, then it is unlikely we would have observed $k$ or fewer failures in the first place. This gives us confidence that the true number of failures, $K$, is strictly less than $K^\prime$.

Pushing this reasoning to its natural limit, we therefore seek the smallest value $K^\prime$ for which $p(k, K^\prime; n, N) \lt \alpha$. We will use $K^\prime - 1$ for the $1-\alpha$ upper confidence limit on $K$. Equivalently we could maximize the value $K^{\prime\prime}$ for which $p(k, K^{\prime\prime}; n, N) \ge \alpha$:

$$\operatorname{UCL}_\alpha(k) = \max\{K\,|\, p(k, K; n, N) \ge \alpha\}.\tag{1}$$

Now for the details. The chance of observing exactly $k$ failures is the chance that (a) our $n$-element sample contains those $k$ failures and (b) the remaining $N-n$ members of the population contain the remaining $K-k$ failures. This describes $\binom{K}{k} \binom{N-K}{n-k}$ subsets out of $\binom{N}{n}$ equally likely subsets. Summing these for all values from $k=0$ to $k$ equal to the actual number of observed failures gives

$$p(k,K;n,N) = \frac{1}{\binom{N}{n}}\sum_{j=0}^k \binom{K}{j} \binom{N-K}{n-j}.$$

This is the Hypergeometric distribution.

In R, for instance, the parameters to supply to the Hypergeometric functions are $N-K$ (called m on the manual page), $K$, (n), and $n$ (k). The phyper function implements $p$ and the qhyper function implements its inverse.

Take, for example, a case of a population with $N=8$ elements from which a sample of size $n=4$ is drawn and $k=1$ failure is observed. Then

$$p(3, K, 4, 8) = \frac{1}{\binom{8}{4}}\sum_{j=0}^1 \binom{K}{j}\binom{8-K}{4-j} = \frac{1}{70}\left(\binom{8-K}{4} + K\binom{8-K}{3}\right).$$

The possible values of $K$ range from a minimum of $k=1$ (the one failure observed) to $k = k + (N-n) = 5$ (occurring when every non-observed member of the population is a failure). Plugging these values into the preceding equation gives the sequence

$$(70, 55, 35, 17, 5)/70 \approx (100, 79, 50, 24, 7)/100.$$

R will compute them in one stroke as

phyper(1, 1:5, 8-(1:5), 4)

We read these numbers like this:

  • There is $100\%$ confidence the population has at least $K=1$ failure. (We have seen it.)

  • There is $79\%$ confidence the population has at least $K=2$ failures. In other words, we attach high confidence to the existence of at least one more failure in the $N-n=4$ members not observed.

  • There is $50\%$ confidence the population has at least $K=3$ failures. This may seem counter-intuitive: since we have seen half the population and observed $k=1$ failure, shouldn't we assign exactly $1/2=50\%$ confidence to seeing one more failure in the other half of the population? This is where confidence differs from probability. The correct approach asks this question: when there are $K=3$ failures in the population (of size $N=8$) and we sample half of it, what is the chance we will see just zero or one failures? By symmetry--the unsampled members themselves constitute a random sample of $N-n=4$, too--this is the chance that the remaining unsampled members will consist of just zero or one failures. Thus observing zero or one failures out of three in the population is an event that will occur half the time. Consequently, actually observing one failure is perfectly consistent with the presence of three failures total.

  • There is $24\%$ confidence the population has at least $K=4$ failures and $7\%$ confidence it has at least $K=5$ failures. These numbers are starting to get close to typical values of $\alpha$. For instance, with $\alpha=0.10$ the $90\%$ upper confidence limit for $K$ would be $K=4$. But with $\alpha=0.05$ the upper $95\%$ UCL for $K$ is $K=5$. When we observe one failure out of four, in a sample from a population of eight, there is an appreciable risk that all the unsampled members are failures! This is because when five of eight members are failures, there's still a considerable chance--more than $7\%$--that our sample just happens to include all three successes.

Note that qhyper in R does not compute confidence limits. You need to search, just as we did in this example. A brute-force search (but relatively efficient for R) tests all the values, as in

which(phyper(1, 1:5, 8-(1:5), 4) >= .10)

This command returns the indexes 1 2 3 4, showing that the first four elements of the vector 1:5 (representing the possible values of $K$) are consistent with our observations at the $\alpha=0.10$ level. The largest of those, $4$, corresponds to $K=4$ as we found through inspection.

In the example of the question, a sample of size $n=100$ is taken from a population of $N=500$ and $k=3$ failures are observed. What is a $90\%$ upper confidence limit for the total number of failures $K$? The R search is

`max(which(phyper(3, 1:100, 500-(1:100), 100) >= .10))`

(The correspondence between this and the mathematical formula for the UCL in $(1)$ is evident.)

It returns a UCL of $30$. Let's double-check by computing the probabilities $p(3,30;100,500)$ and $p(3,31;100,500)$. The first should exceed $10\%$ and the second should drop just below it:

> phyper(3, 30, 500-30, 100)
[1] 0.1151626
> phyper(3, 31, 500-31, 100)
[1] 0.09959309

That's exactly what happens. We conclude, with at least $90\%$ confidence, that there exist up to (but no more than) $K-k=30-3=27$ additional failures among the $N-n=500-100=400$ unsampled members of the population.

  • 1
    $\begingroup$ As always, thank you for the time and such a detailed response, @whuber. This is precisely what I needed. My "ah ha" moment was during your second bullet point when the population was split in half. $\endgroup$
    – Ashe
    Apr 18, 2016 at 20:57
  • $\begingroup$ I've been playing with the result above, but it seems to break down when you have a really high failure rate. Take the example of a population of 8 and you sample 4. If all of those 4 fail, then the equation above tells you will 100% probability that all member of the population are faulty. This seems like a "bug". $\endgroup$
    – nebcb
    Jun 1, 2021 at 2:20
  • 1
    $\begingroup$ @nebcb: The confidence isn't the probability that all members of the population are faulty. If there are 4 (5, 6, 7, or 8) failures in the population of 8, what is the chance we observe 4 or fewer failures in the sample of 4? The answer is 100% in every case, and an upper confidence limit at the 100% (or any lower) level is 8. It'd be a weird procedure that, given an entirely faulty sample, ruled out an entirely faulty population at any confidence level. Not a bug but a feature. $\endgroup$ Jun 1, 2021 at 9:00

Not the answer you're looking for? Browse other questions tagged or ask your own question.