How to properly set up hypothesis testing in an operations reporting environment

Question

I am in charge of quality control of n = 400 reports. Most of the quality checks are automated, but some have to be manual. I need to figure out how many reports my team needs to check manually in order to be fairly confident that all of the reports are error free in the areas which require manual checks. How can I correctly use hypothesis testing to answer this problem?

My first instinct is to start with something like this:

Null hypothesis: at least one report contains errors.
Alternate hypothesis: all n reports are error free.

To collect data, we can check k reports manually and find numErrors errors in those k reports. Then,

If numErrors > 0:
    Null hypothesis is true with 100% probability
Elif numErrors == 0:
    Alternate hypothesis is true with x% probability

What I am struggling with is how to figure out the optimal value of k, and once we have k, how to determine the x% probability the Alternate Hypothesis is true if numErrors == 0. Any ideas? Thanks!

Answer 1

A natural way of solving this problem is by means of Bayesian inference. Given that we have checked a small sample $k$ out of $n$ reports, and we have not found a single error, what is the probability that not a single report ($\tilde{k}=0$) out of all remaining $\tilde{n}=400-n$ reports contains an error?

Let us begin by assuming that errors are i.i.d. and that, a priori, we do not know anything about the probability $\pi$ of a report to be erroneous. Thus, our prior uncertainty is $p(\pi) = U(\pi \mid 0,1)$, i.e., a uniform distribution over the interval $[0,1]$. Now, given the observation that $k$ out of $n$ samples contained errors, we can update our knowledge about $\pi$ and obtain the posterior density $p(\pi \mid k,n) = \textrm{Beta}(\pi \mid k+1,n-k+1)$. Given this posterior, we can further ask about the predictive density over erroneous reports in a new sample, i.e., in the remaining $\tilde{n}=400-n$ reports. This density is $p(\tilde{k} \mid \tilde{n},k,n) = \textrm{Bb}(\tilde{k} \mid \tilde{n},k+1,n-k+1)$, where $\textrm{Bb}(\cdot)$ denotes the Beta-binomial distribution.

We can now use the above result to answer the original question.

Given $k=0$ erroneous reports in our manual sample, what is the probability that there is not a single ($\tilde{k}=0$) erroneous report among the remaining $\tilde{n}=400-n$ reports? The answer is $\textrm{Bb}(0 \mid 400-n,1,n+1)$. It may come as a surprise that if the manual sample contains, say, $n=20$ reports, then the probabiity of having zero faulty reports in the remaining 380 reports is no larger than $5.23\%$ (see Figure below).

Conversely, in order to obtain a $95\%$ posterior probability that there are zero faulty reports in the remaining set, we would have to manually check no less than $380$ reports, since $\textrm{Bb}(0,400-380,1,381) = 0.9501$.

Note that the above assumes a flat prior over the error probability $\pi$. If one can safely assume a prior with a mean closer to 0, the number of required tests would shrink quickly. For example, one could use the posterior $p(\pi \mid k,n)$ obtained on a previous dataset as a prior for the current one. The above solution strategy would remain unchanged.