In many classification problems, the cost of a FP is different from the cost of a FN. In spam detection, a FP (a regular email classified as spam) should have a high cost. In cancer prediction, a FN (a cancer goes undetected) should have a high cost.
In spam detection for example, the cost of a FP is high so a natural objective is to limit their number, while minimizing the number of FN. If we translate this idea using precision and recall, we want to maximize precision while recall > T (for some high T). In other words, we are interested in the partial AUC (where recall > T) rather than the (full) AUC.
However in practice, the concept of partial AUC seems to attract little attention:
- popular data science libraries (such as scikit) do not implement it: https://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics
- the most cited papers about cancer prediction and spam prediction use accuracy or the (full) AUC, but not the partial AUC
Isn't the partial AUC a good metric to compare classifiers on cost-sensitive classification problems? Isn't the AUC sub-optimal? Or am i missing something that makes these considerations irrelevant?