My specific context is detecting errors in automatic classification. Want to regularly manually review a subset of all events in one class, to ensure that the process putting them in that class is highly reliable. I’m wondering how few events I can sample and still get a solid estimate of reliability at the population level.
The method I’ve come across which so far seems most promising is Cochran’s formula. However, if I’m doing the math correctly it gives some results which seem very counterintuitive if not obviously wrong. For example, assuming 2% incidence of error and requiring 95% confidence in the conclusions, I calculate a sample size of 20 as follows.
\begin{align}
n &= \frac{Z^2(p)(q)}{e^2}\\&=\frac{1.6^2(0.98) (.02) }{0.05^2}\\&= 20.
\end{align}
Hard to believe that you could be highly confident about a 2% incidence in population with a sample size of 20. Using a confidence level of 98% gives a more reasonable sample size of ~200, but the tiny sample size suggested for 95% confidence makes me wonder about the method – say if there are some prerequisite assumptions for using it that I’m not aware of.
In case it's relevant, the population size in this case is tens of thousands.
Tentative plan is to use a Z Proportion Hypothesis Test on the sample after it has been manually labeledlabelled, to flag instances where we can be confident that error rate is above 2%. More fundamentally, I want to be confident that error rate is below 2% in population when less than 2% error is found in sample – that is the key requirement on sample size.
Should I use Cochran’s formula to find that sample size? Or is there a better method for this case?
