# Devising an acceptance sampling plan for False Negative Rate

I need to evaluate a binary classifier that classifies inputs in positives and negatives. Since all predicted positives (PP) are assessed, I have complete data on the true positives (TP) and the false positives (FP). However, predicted negatives (PN) are sampled and I only have sample-based estimates on the false negatives (FN) and true negatives (TN).

I'm looking for (or most likely, I have to devise) an acceptance sampling plan that ensures that the classifier FNR is below a certain threshold. This plan would have as parameters

• the lot (or population) size: in my case $$PN=FN+TN$$
• the target metric limit that has to be achieved: in my case $$FNR < 0.05$$
• a sample size $$n$$

and for those, it would define

• a cutoff threshold $$c$$ such that, said $$x$$ the number of false negatives in the sample, the lot is accepted if $$x \leq c$$ and rejected otherwise

Ultimately, the sampling plan provides can estimate the probability that a certain lot is accepted ($$x \leq c$$) while it should have been rejected ($$FNR \geq 0.05$$). Typically, this conditional probability has to be below a certain threshold, in my case 10%:

$$P(x \leq c ~|~ FNR \geq 0.05) < 0.1$$

How can I find this value for $$c$$? I've never seen sampling plans targeted at measuring machine learning metrics. I feel the answer lies along the lines of using the hypergeometric distribution to estimate this probability, but I can't see the math quite yet.

Update: this is the furthest I managed to reach after some additional tinkering. From the theory, we know that a random variable $$X$$ following the hypergeometric distribution $$X \sim H(N,K,n)$$ models drawing $$n$$ items without replacement from a population of $$N$$, where $$K \leq N$$ are of interest. In my case, $$N=TN+FN$$ (the whole population where to sample) and $$K=FN$$ (the whole number of items of interest in N).

Given that $$FNR=\frac{FN}{FN+TP}$$, we have that $$FN=\frac{TP \cdot FNR}{1-FNR}$$. Note that I can calculate $$FN$$ if I assume $$FNR = 0.05$$.

Essentially, this means that I can use the hypergeometric distribution with parameters $$N=PN=TN+FN$$ and $$K=FN=\frac{TP \cdot FNR}{1-FNR}$$ to calculate some pairs $$(n, c)$$ for $$P(X \leq c)$$, i.e., the probability of having less than $$c$$ out of $$n$$ false negatives when sampling among $$N$$ items with $$K$$ total false negatives (which contribute to have a 5% FNR).

Is this logic going in the right direction?