ROC and AUC for imbalanced data? [duplicate]

Question

I've some trouble understanding how to interpret the ROC and it's area under the curve for a classification task.

In general, the higher the AUC the better the model can classify true as true and false as false, independently from the threshold value, correct?

If AUC = 1 you can say that there is a threshold where True positiv rate (Recall) is 100%, meaning all true observations are predicted as true and False Positive Rate is zero, meaning that there is no predicted true value that is actually false. Moreover the TPR is always 100% for every threshold, only the FPR increases. Is this correct?

What about imbalanced data? So if I have 95 data points of class 1 and 5 of class 2 and my classifier always predicts class 1, i would still have a accuracy of 95%. So I understand why the accuracy is not good for that case. But what about the AUC? Is it meaningful if I have imbalanced data? Would this classifier achieve a high or low AUC value? Does it matter which class i choose to be the "true" one?

I looked through some blogs. Some say yes, some say no. Can anyone explain it with an example for a better understanding?

Thank you! Cheers!

Answer 1

In general, the higher the AUC the better the model can classify true as true and false as false, independently from the threshold value, correct?

No. If two ROCs cross, the ROC with the higher AUC will have at least a measurable subset of thresholds where ROC with inferior AUC is a better classifier.

What about imbalanced data? So if I have 95 data points of class 1 and 5 of class 2 and my classifier always predicts class 1, i would still have a accuracy of 95%.

Since the AUC is a measure of recall, it doesn't matter if the design is imbalanced, or even if samples were determined from outcome dependent sampling. The only thing to be affected are the confidence limits of the AUC (and for any given threshold-determined sensitivity and specificity).

If your real interest lies in positive/negative predictive value (PPV/NPV), you should just report those values and be done. Remember for diagnostics the critical distinction of $P(\text{Disease} | \text{Test positive})$ vs $P(\text{Test positive} | \text{Disease})$.

Yes it's true that, for these metrics, if your non-diseased subset is 95% of the sample, then an "always no" classifier is going to send the 95% who are negative straight home without any waiting, without any labs, and without any bills. That 95% NPV you get for free. 0% PPV means the other 5% go home with progressing disease severity. The real way to determine the right threshold is quantifying the costs and consequences of your incorrect decisions.

Answer 2

In general, the higher the AUC the better the model can classify true as true and false as false, independently from the threshold value, correct?

No. Here's a simple counterexample. Imagine you have 1 positive and 9 negative samples. Model A predicts the positive sample at 5% and all others at less than 1%, giving it 100% AUC. Model B predicts one negative sample at 80%, the positive sample at 75%, and all other negative samples at less than 5%, giving it an AUC of 89%. At any threshold above 5%, Model B is the better classifier.

If AUC = 1 you can say that there is a threshold where True positiv rate (Recall) is 100%, meaning all true observations are predicted as true and False Positive Rate is zero, meaning that there is no predicted true value that is actually false. Moreover the TPR is always 100% for every threshold, only the FPR increases. Is this correct?

Yes, for all thresholds where the TPR is defined. I guess that may have been implied in your statement, but I put it explicitly just in case. For a threshold that's higher than any of the predicted probabilities, the TPR is clearly undefined, because its denominator is zero. But for all thresholds at or below the highest predicted probability, you're right.

What about imbalanced data? So if I have 95 data points of class 1 and 5 of class 2 and my classifier always predicts class 1, i would still have a accuracy of 95%. So I understand why the accuracy is not good for that case. But what about the AUC? Is it meaningful if I have imbalanced data? Would this classifier achieve a high or low AUC value?

You don't have to worry about getting a "good" result for something as trivial as predicting all ones. If you do that, your AUC should be 50% on average. Think of it this way. If you have a string of $0$s and $1$s, and you sort them uniformly at random, then on average, the $1$s will be distributed uniformly throughout the string, no matter what proportion it's in. Thus the AUC will, on average, be 50%. Of course, the smaller your sample size, the more likely AUC is to deviate from 50% for such a prediction. But not in a fixed bias way, like accuracy. By random chance, your AUC can be misleadingly high, or misleadingly low with equal probability if you predict all ones with a small sample of ones.

One way in which AUC can be misleadingly high is if you have imbalanced data in favor of $0$s, and a model with high recall but low precision, as I explained here. This is forgivable if you are more tolerant of false positives than false negatives.