8
$\begingroup$

I just finished reading this discussion. They argue that PR AUC is better than ROC AUC on imbalanced dataset.

For example, we have 10 samples in test dataset. 9 samples are positive and 1 is negative. We have a terrible model which predicts everything positive. Thus, we will have a metric that TP = 9, FP = 1, TN = 0, FN = 0.

Then, Precision = 0.9, Recall = 1.0. The precision and recall are both very high, but we have a poor classifier.

On the other hand, TPR = TP/(TP+FN) = 1.0, FPR = FP/(FP+TN) = 1.0. Because the FPR is very high, we can identify that this is not a good classifier.

Clearly, ROC is better than PR on imbalanced datasets. Can somebody explain why PR is better?

$\endgroup$
  • $\begingroup$ Precision and Recall both ignore False Negatives. The usual justification for using PR tradeoff (curves or F-score) is that the number of Negatives and False Negatives is huge relative to TP and FP. So TNR->1 and FPR->0 (sum to 1 with same |Negs| denominator). So PR in this case does reflect (amplify or zoom in on) the trade off TP vs FP, but this is not meaningful and what is relevant is an increase in the Youden J index (Informedness=TPR-FPR=TPR+TNR-1=Sensitivity+Specificity-1) which corresponds to twice the area between the triangular single operating point curve and the ROC chance line. $\endgroup$ – David M W Powers Nov 21 '17 at 13:13
  • $\begingroup$ @DavidMWPowers, why not turn that into an official answer? That seems like a very informative response to me. $\endgroup$ – gung Oct 10 '18 at 14:13
  • $\begingroup$ Precision, recall, sensitivity, and specificity are improper discontinuous arbitrary information-losing accuracy scores and should not be used. They can be especially problematic under imbalance. The $c$-index (concordance probability; AUROC) works fine under extreme balance. Better: use a proper accuracy scoring rule related to log-likelihood or the Brier score. $\endgroup$ – Frank Harrell Oct 10 '18 at 15:21
1
$\begingroup$

First, the claim on the Kaggle post is bogus. The paper they reference, "The Relationship Between Precision-Recall and ROC Curves", never claims that PR AUC is better than ROC AUC. They simply compare their properties, without judging their value.

ROC curves can sometimes be misleading in some very imbalanced applications. A ROC curve can still look pretty good (ie better than random) while misclassifying most or all of the minority class.

In contrast, PR curves are specifically tailored for the detection of rare events and are pretty useful in those scenarios. But they don't translate well to more balanced cases, or cases where negatives are rare.

As always, it is important to understand the tools that are available to you and select the right one for the right application. I suggest reading the question ROC vs precision-and-recall curves here on CV.

$\endgroup$
0
$\begingroup$

Your example is definitely correct.

However, I think in the context of Kaggle competition / real life application, a skewed dataset usually means a dataset with much less positive samples than negative samples. Only in this case, PR AUC is more "meaningful" than ROC AUC.

Consider a detector with TP=9, FN=1, TN=900, FP=90, where there are 10 positive and 990 negative sample. TPR=0.9, FPR=0.1 which indicates a good ROC score, however Precision=0.1 which indicates a bad PR score.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.