Given a classifier $f$, $N$ possible classes, and an input $x$, $f$ produces a class from $[1,..., N]$ and its matching confidence $[0,...,100]$. Then I run $f$ on a large set of examples $X$, and I have the empirical results. The goal is to select N thresholds, $[t_1,...,t_N]$ such that $f$ on $X$ maximizes recall under constrained precision.

Details: For each class C and $x$, a specific example of class C:

  • the True Positives $TP$ are the amount of examples $x$, such that $f$($x$) gives class $c$ with confidence >= $t_C$.
  • the False Positives $FP$ are the amount in which $f$($x$) gives a class different from $c$ with confidence >= $T_C$.
  • The False Negatives $FN$ are the amount in which $f$($x$) gives any class with confidence < $T_C$.

The recall is the sum over all classes of $\frac{TP}{TP+FN}$, and precision is the sum over all classes of $\frac{TP}{TP+FP}$.

In other words, we select many thresholds to optimize a single global recall score under constrained global precision, i.e., maximize the number of found examples under the constraint of 90% precision across all seen examples.

I looked at convex optimization, but Recall/Precision are non-convex; I'm not sure how to proceed from here.




Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.