Isn't partial AUC a better metric than AUC for cost-sensitive classification problems?

Question

In many classification problems, the cost of a FP is different from the cost of a FN. In spam detection, a FP (a regular email classified as spam) should have a high cost. In cancer prediction, a FN (a cancer goes undetected) should have a high cost.

In spam detection for example, the cost of a FP is high so a natural objective is to limit their number, while minimizing the number of FN. If we translate this idea using precision and recall, we want to maximize precision while recall > T (for some high T). In other words, we are interested in the partial AUC (where recall > T) rather than the (full) AUC.

However in practice, the concept of partial AUC seems to attract little attention:

• popular data science libraries (such as scikit) do not implement it: https://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics
• the most cited papers about cancer prediction and spam prediction use accuracy or the (full) AUC, but not the partial AUC
  • https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=spam+detection&btnG=&oq=spam+
  • https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=cancer+prediction&btnG=

Isn't the partial AUC a good metric to compare classifiers on cost-sensitive classification problems? Isn't the AUC sub-optimal? Or am i missing something that makes these considerations irrelevant?

Answer 1

Or am i missing something that makes these considerations irrelevant?

My impulse would be to go with this one.

For the partial AUC, you need to specify an acceptable recall, then calculate partial AUC and tune or select your prediction model. Thus, the optimal model can depend on your acceptable recall. At first glance, this might make sense - if we change what is "acceptable", of course we get different results, right?

At second glance, not so much: what we really want is a probabilistic classification for probabilities of spam or cancer, which we can then subject to different thresholds for deciding on actions, and we may well have more than two possible actions even if there are only two classes. The important thing is that the predictive possibilities should be calibrated: if our model says an email has a probability $\hat{p}$ to be spam, then ideally, that probability should be correct. But using partial AUC means that the predicted probabilities will be conditional on a particular parameter, namely the threshold that we use in assessing whether a given model reaches acceptable recall, and we can (and need to) tune this threshold. There is no reason to expect that any model that reaches acceptable recall based on a tuned threshold should be calibrated.

Thus, my recommendation would be to try to find a calibrated model, by assessing probabilistic predictions using proper scoring rules. Of course, we might be interested in different tails of the predicted probabilities, which also ties into how much information we have and how well we can predict. In this vein, Gneiting & Ranjan (2011, JASA) discuss weighted scoring rules for when we want to emphasize regions of interest, like the center or tails of the probability density, without losing propriety.

Answer 2

I agree with you that the ROC AUC can be very misleading - a model with a high AUC can have a large number of false positives, which as you mentioned can be very undesirable in many circumstances.

popular data science libraries (such as scikit) do not implement it: https://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics

Scikit-learn does have an implementation of partial AUC. You calculate the ROC AUC using sklearn.metrics.roc_auc_score and specify the desired maximum false positive rate using the max_fpr parameter. This will give you the AUC only within a certain range a false positive rate between 0 and the maximum you specify.

e.g., partial_auc = roc_auc_score(y_true, y_pred, max_fpr=0.05)

Answer 3

You can rationalize the asymmetric FP and FN mistakes using a loss function

$$\ell(y,f(x)) = 1\{y=1,f(x)=0\} + c1\{y=0,f(x)=1\},$$

where $c$ is the cost of the FP mistake. Then it is natural to measure the performance of the classifier using the out-of-sample losses $$\frac{1}{n}\sum_{i=1}^n\ell(y_i,\hat f(x_i))$$

I think that it makes much more sense than the AUC and its derivatives, see https://arxiv.org/pdf/2010.08463.pdf