I just finished reading this discussion. They argue that PR AUC is better than ROC AUC on imbalanced dataset.

For example, we have 10 samples in test dataset. 9 samples are positive and 1 is negative. We have a terrible model which predicts everything positive. Thus, we will have a metric that TP = 9, FP = 1, TN = 0, FN = 0.

Then, Precision = 0.9, Recall = 1.0. The precision and recall are both very high, but we have a poor classifier.

On the other hand, TPR = TP/(TP+FN) = 1.0, FPR = FP/(FP+TN) = 1.0. Because the FPR is very high, we can identify that this is not a good classifier.

Clearly, ROC is better than PR on imbalanced datasets. Can somebody explain why PR is better?

  • 1
    $\begingroup$ Precision and Recall both ignore False Negatives. The usual justification for using PR tradeoff (curves or F-score) is that the number of Negatives and False Negatives is huge relative to TP and FP. So TNR->1 and FPR->0 (sum to 1 with same |Negs| denominator). So PR in this case does reflect (amplify or zoom in on) the trade off TP vs FP, but this is not meaningful and what is relevant is an increase in the Youden J index (Informedness=TPR-FPR=TPR+TNR-1=Sensitivity+Specificity-1) which corresponds to twice the area between the triangular single operating point curve and the ROC chance line. $\endgroup$ Nov 21, 2017 at 13:13
  • 2
    $\begingroup$ @DavidMWPowers, why not turn that into an official answer? That seems like a very informative response to me. $\endgroup$ Oct 10, 2018 at 14:13
  • 4
    $\begingroup$ Precision, recall, sensitivity, and specificity are improper discontinuous arbitrary information-losing accuracy scores and should not be used. They can be especially problematic under imbalance. The $c$-index (concordance probability; AUROC) works fine under extreme balance. Better: use a proper accuracy scoring rule related to log-likelihood or the Brier score. $\endgroup$ Oct 10, 2018 at 15:21

3 Answers 3


First, the claim on the Kaggle post is bogus. The paper they reference, "The Relationship Between Precision-Recall and ROC Curves", never claims that PR AUC is better than ROC AUC. They simply compare their properties, without judging their value.

ROC curves can sometimes be misleading in some very imbalanced applications. A ROC curve can still look pretty good (ie better than random) while misclassifying most or all of the minority class.

In contrast, PR curves are specifically tailored for the detection of rare events and are pretty useful in those scenarios. They will show that your classifier has a low performance if it is misclassifying most or all of the minority class. But they don't translate well to more balanced cases, or cases where negatives are rare.

In addition, because they are sensitive to the baseline probability of positive events, they don't generalize well and only apply to the specific dataset they were built on, or to datastets with the exact same balance. This means it is generally difficult to compare PR curves from different studies, limiting their usefulness.

As always, it is important to understand the tools that are available to you and select the right one for the right application. I suggest reading the question ROC vs precision-and-recall curves here on CV.


Your example is definitely correct.

However, I think in the context of Kaggle competition / real life application, a skewed dataset usually means a dataset with much less positive samples than negative samples. Only in this case, PR AUC is more "meaningful" than ROC AUC.

Consider a detector with TP=9, FN=1, TN=900, FP=90, where there are 10 positive and 990 negative sample. TPR=0.9, FPR=0.1 which indicates a good ROC score, however Precision=0.1 which indicates a bad PR score.


You're half way there.

Usually when I do imbalanced models, heck, even balanced models, I look at PR for ALL my classes.

In your example, yes, your positive class has P = 0.9 and R = 1.0. But what you should look at are ALL your classes. So for your negative class, your P = 0 and your R = 0. And you usually don't just look at PR scores individually. You want to look at F1-score (F1 macro or F1 micro, depending on your problem) that is a harmonic average of your PR scores for both class 1 and class 0. Your class 1 PR score is super good, but combine that with your class 0 PR score, your F1-score will be TERRIBLE, which is the correct conclusion for your scenario.

TL,DR: Look at PR scores for ALL your classes, and combine them with a metric like F1-score to have a realistic conclusion about your model performance. The F1-score for your scenario will be TERRIBLE, which is the correct conclusion for your scenario.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.