I need to evaluate a binary classifier that classifies inputs in positives and negatives. Since all predicted positives (PP) are assessed, I have complete data on the true positives (TP) and the false positives (FP). However, predicted negatives (PN) are sampled and I only have sample-based estimates on the false negatives (FN) and true negatives (TN).

I'm looking for (or most likely, I have to devise) an acceptance sampling plan that ensures that the classifier FNR is below a certain threshold. This plan would have as parameters

  • the lot (or population) size: in my case $PN=FN+TN$
  • the target metric limit that has to be achieved: in my case $FNR < 0.05$
  • a sample size $n$

and for those, it would define

  • a cutoff threshold $c$ such that, said $x$ the number of false negatives in the sample, the lot is accepted if $x \leq c$ and rejected otherwise

Ultimately, the sampling plan provides can estimate the probability that a certain lot is accepted ($x \leq c$) while it should have been rejected ($FNR \geq 0.05$). Typically, this conditional probability has to be below a certain threshold, in my case 10%:

$P(x \leq c ~|~ FNR \geq 0.05) < 0.1$

How can I find this value for $c$? I've never seen sampling plans targeted at measuring machine learning metrics. I feel the answer lies along the lines of using the hypergeometric distribution to estimate this probability, but I can't see the math quite yet.

Update: this is the furthest I managed to reach after some additional tinkering. From the theory, we know that a random variable $X$ following the hypergeometric distribution $X \sim H(N,K,n)$ models drawing $n$ items without replacement from a population of $N$, where $K \leq N$ are of interest. In my case, $N=TN+FN$ (the whole population where to sample) and $K=FN$ (the whole number of items of interest in N).

Given that $FNR=\frac{FN}{FN+TP}$, we have that $FN=\frac{TP \cdot FNR}{1-FNR}$. Note that I can calculate $FN$ if I assume $FNR = 0.05$.

Essentially, this means that I can use the hypergeometric distribution with parameters $N=PN=TN+FN$ and $K=FN=\frac{TP \cdot FNR}{1-FNR}$ to calculate some pairs $(n, c)$ for $P(X \leq c)$, i.e., the probability of having less than $c$ out of $n$ false negatives when sampling among $N$ items with $K$ total false negatives (which contribute to have a 5% FNR).

Is this logic going in the right direction?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.