determining the sample size for quality test

Question

It has been a long time since I did statistics in uni and I have a "simple problem" that I need solving.

I need to determin for a sample size for a limited population of documents ( lets say 200 ). The sample size should answer the question: If all of the documents are correct ( yes/no ) then with 99% probabilty, or confidance, I can say that all the documents in the population are correct.

Answer 1

I originally thought this could be done with a binomial approximation, but I've since changed my mind.

Because you have a finite sample (N=200 documents) then we can do Bayesian inference to determine the probability no documents have errors given we observe our entire sample is error free. This question asks something similar and offers the posterior distribution.

Following Tim's answer in the linked question, let's make our "target" state an incorrect documents and determine the probability there are 0 incorrect documents given we have only observed correct documents thus far.

Using this model, it looks like you'd have to sample almost all documents before there is a 99% probability that all documents are correct, which means you might as well sample them all.

Answer 2

We have $N$ documents, $E$ of those documents have errors. We don't know how many errors that there are, but if we assume that these errors may have been introduced randomly with probability $p$ for each of the documents, then our prior on the number of errors $E$ would follow the binomial distribution. However, we also don't know the rate $p$ at which errors might have been introduced. Given that we're not even sure apriori whether errors are possible, or whether documents without errors are possible, it would be sensible to use a Jeffreys prior on $p$ to be as ignorant as possible. Then, we sample $n$ of the documents, assess each of those, and then we identify that there are $e$ errors amongst those documents. The likelihood of observing $e$ errors can be described by the hypergeometric distribution. Overall, our model can be illustrated as follows:

$$ \begin{aligned} p &\sim \mathrm{Beta}(\frac{1}{2}, \frac{1}{2}) \\ E &\sim \mathrm{Binomial}(N, p) \\ e &\sim \mathrm{Hypergeometric}(n, E, N - E) \\ \end{aligned} $$

Using this model, we evaluate the posterior probability distribution for the number of errors $P(E|n,e,N)$, which we can evaluate analytically in this case to be:

$$ \begin{aligned} P(E|n,e, N) &= \frac{P(E|N) P(n, e| E, N)}{P(n, e | N)} \\ &= \frac{1}{Z}\binom{N}{E}\binom{E}{e} \binom{N - E}{n - e} B(E + 1/2, N - E + 1/2) \\ \end{aligned} $$

Where $B(x,y)$ is the beta function, and $Z$ is a normalization constant to ensure that $\sum_E P(E|n,e, N) = 1$.

The following is a chart that displays the probability that there are any errors at all, $1 - P(E=0|n, e=0, N=200)$, as a function of $n$, assuming that we never observe any documents with errors, $e = 0$ as we continue to sample the documents, in the case when there are $N = 200$ documents.

import numpy as np
from scipy.special import beta, binom
import matplotlib.pyplot as plt


def p(E, N, n, e):
    p = np.array([
        binom(N, E) * binom(E, e) * binom(N - E, n - e) * beta(E + 1/2, N - E + 1/2)
        for E in range(0, N + 1)
    ])
    return p[E] / p.sum()


N = 200
ns = np.arange(0, N + 1)
p_any_errors = np.array([1 - p(E=0, N=N, n=n, e=0) for n in ns])

target_probability = 0.01
target_documents_sampled = np.interp(target_probability, p_any_errors[::-1], ns[::-1])

plt.plot(ns, p_any_errors)
plt.yscale('log')
xlim, ylim = plt.xlim(), plt.ylim()
plt.hlines([target_probability], xlim[0], xlim[1])
plt.vlines([target_documents_sampled], ylim[0], ylim[1])
plt.xlim(xlim), plt.ylim(ylim)
plt.xlabel('number of documents sampled')
plt.ylabel('probability of any errors')
plt.title(
    f"sample {target_documents_sampled:0.0f} documents "
    f"to reach {target_probability} probability of no errors"
)

Using this model, we would require sampling about 196 out of the 200 documents to reach a probability of 0.01 that there are no errors in the full set of 200 documents. So, it seems that if we want to reach this probability threshold, we can't really save much work evaluating the documents relative to evaluating all of the documents.