Given a classifier $f$, $N$ possible classes, and an input $x$, $f$ produces a class from $[1,..., N]$ and its matching confidence $[0,...,100]$. Then I run $f$ on a large set of examples $X$, and I have the empirical results. The goal is to select N thresholds, $[t_1,...,t_N]$ such that $f$ on $X$ maximizes recall under constrained precision.
Details: For each class C and $x$, a specific example of class C:
- the True Positives $TP$ are the amount of examples $x$, such that $f$($x$) gives class $c$ with confidence >= $t_C$.
- the False Positives $FP$ are the amount in which $f$($x$) gives a class different from $c$ with confidence >= $T_C$.
- The False Negatives $FN$ are the amount in which $f$($x$) gives any class with confidence < $T_C$.
The recall is the sum over all classes of $\frac{TP}{TP+FN}$, and precision is the sum over all classes of $\frac{TP}{TP+FP}$.
In other words, we select many thresholds to optimize a single global recall score under constrained global precision, i.e., maximize the number of found examples under the constraint of 90% precision across all seen examples.
I looked at convex optimization, but Recall/Precision are non-convex; I'm not sure how to proceed from here.
TIA
