1
$\begingroup$

I need to evaluate a binary classifier that classifies inputs in positives and negatives. Since all predicted positives (PP) are assessed, I have complete data on the true positives (TP) and the false positives (FP). However, predicted negatives (PN) are sampled and I only have sample-based estimates on the false negatives (FN) and true negatives (TN).

I'm looking for (or most likely, I have to devise) an acceptance sampling plan that ensures that the classifier FNR is below a certain threshold. This plan would have as parameters

  • the lot (or population) size: in my case $PN=FN+TN$
  • the target metric limit that has to be achieved: in my case $FNR < 0.05$
  • a sample size $n$

and for those, it would define

  • a cutoff threshold $c$ such that, said $x$ the number of false negatives in the sample, the lot is accepted if $x \leq c$ and rejected otherwise

Ultimately, the sampling plan provides can estimate the probability that a certain lot is accepted ($x \leq c$) while it should have been rejected ($FNR \geq 0.05$). Typically, this conditional probability has to be below a certain threshold, in my case 10%:

$P(x \leq c ~|~ FNR \geq 0.05) < 0.1$

How can I find this value for $c$? I've never seen sampling plans targeted at measuring machine learning metrics. I feel the answer lies along the lines of using the hypergeometric distribution to estimate this probability, but I can't see the math quite yet.


Update: this is the furthest I managed to reach after some additional tinkering. From the theory, we know that a random variable $X$ following the hypergeometric distribution $X \sim H(N,K,n)$ models drawing $n$ items without replacement from a population of $N$, where $K \leq N$ are of interest. In my case, $N=TN+FN$ (the whole population where to sample) and $K=FN$ (the whole number of items of interest in N).

Given that $FNR=\frac{FN}{FN+TP}$, we have that $FN=\frac{TP \cdot FNR}{1-FNR}$. Note that I can calculate $FN$ if I assume $FNR = 0.05$.

Essentially, this means that I can use the hypergeometric distribution with parameters $N=PN=TN+FN$ and $K=FN=\frac{TP \cdot FNR}{1-FNR}$ to calculate some pairs $(n, c)$ for $P(X \leq c)$, i.e., the probability of having less than $c$ out of $n$ false negatives when sampling among $N$ items with $K$ total false negatives (which contribute to have a 5% FNR).

Is this logic going in the right direction?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.