41
$\begingroup$

I just finished reading this discussion. They argue that PR AUC is better than ROC AUC on imbalanced dataset.

For example, we have 10 samples in test dataset. 9 samples are positive and 1 is negative. We have a terrible model which predicts everything positive. Thus, we will have a metric that TP = 9, FP = 1, TN = 0, FN = 0.

Then, Precision = 0.9, Recall = 1.0. The precision and recall are both very high, but we have a poor classifier.

On the other hand, TPR = TP/(TP+FN) = 1.0, FPR = FP/(FP+TN) = 1.0. Because the FPR is very high, we can identify that this is not a good classifier.

Clearly, ROC is better than PR on imbalanced datasets. Can somebody explain why PR is better?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Precision and Recall both ignore False Negatives. The usual justification for using PR tradeoff (curves or F-score) is that the number of Negatives and False Negatives is huge relative to TP and FP. So TNR->1 and FPR->0 (sum to 1 with same |Negs| denominator). So PR in this case does reflect (amplify or zoom in on) the trade off TP vs FP, but this is not meaningful and what is relevant is an increase in the Youden J index (Informedness=TPR-FPR=TPR+TNR-1=Sensitivity+Specificity-1) which corresponds to twice the area between the triangular single operating point curve and the ROC chance line. $\endgroup$
    – David M W Powers
    Commented Nov 21, 2017 at 13:13
  • 2
    $\begingroup$ @DavidMWPowers, why not turn that into an official answer? That seems like a very informative response to me. $\endgroup$
    – gung - Reinstate Monica
    Commented Oct 10, 2018 at 14:13
  • 5
    $\begingroup$ Precision, recall, sensitivity, and specificity are improper discontinuous arbitrary information-losing accuracy scores and should not be used. They can be especially problematic under imbalance. The $c$-index (concordance probability; AUROC) works fine under extreme balance. Better: use a proper accuracy scoring rule related to log-likelihood or the Brier score. $\endgroup$
    – Frank Harrell
    Commented Oct 10, 2018 at 15:21
  • $\begingroup$ Your example is artificial, so perhaps you do not realize that it is fundamentally flawed and so probably muddies the clarity of the question. In real life, the minority class is almost always the positive class, not the majority class (we seek to explain exceptions). So, you cannot have 9 positives and 1 negative; you would have 1 positive and 9 negatives. If you reframe the question with that adjustment, then it should be clearer why ROC is not so helpful with your example scenario: TP = 0, FP = 9, TN = 9, FN = 1. Then Precision = 0 and Recall = 0. Clearly a horrible performance. $\endgroup$
    – Tripartio
    Commented Nov 22, 2022 at 8:35

4 Answers 4

Reset to default
35
$\begingroup$

First, the claim on the Kaggle post is bogus. The paper they reference, "The Relationship Between Precision-Recall and ROC Curves", never claims that PR AUC is better than ROC AUC. They simply compare their properties, without judging their value.

ROC curves can sometimes be misleading in some very imbalanced applications. A ROC curve can still look pretty good (ie better than random) while misclassifying most or all of the minority class.

In contrast, PR curves are specifically tailored for the detection of rare events and are pretty useful in those scenarios. They will show that your classifier has a low performance if it is misclassifying most or all of the minority class. But they don't translate well to more balanced cases, or cases where negatives are rare.

In addition, because they are sensitive to the baseline probability of positive events, they don't generalize well and only apply to the specific dataset they were built on, or to datastets with the exact same balance. This means it is generally difficult to compare PR curves from different studies, limiting their usefulness.

As always, it is important to understand the tools that are available to you and select the right one for the right application. I suggest reading the question ROC vs precision-and-recall curves here on CV.

Improve this answer
$\endgroup$
2
  • $\begingroup$ What if we have used weights or a cost matrix to compensate the imbalance? Will it be still be preferred to use the AUC PR? Or using weights or a cost matrix it's enough and then we can use the AUC ROC curve or even the Accuracy as a metric to optimize our model? $\endgroup$
    – skan
    Commented Oct 29, 2022 at 0:33
  • 1
    $\begingroup$ @skan you should ask this as a new question. $\endgroup$
    – Calimo
    Commented Oct 31, 2022 at 16:04
17
$\begingroup$

Your example is definitely correct.

However, I think in the context of Kaggle competition / real life application, a skewed dataset usually means a dataset with much less positive samples than negative samples. Only in this case, PR AUC is more "meaningful" than ROC AUC.

Consider a detector with TP=9, FN=1, TN=900, FP=90, where there are 10 positive and 990 negative sample. TPR=0.9, FPR=0.1 which indicates a good ROC score, however Precision=0.1 which indicates a bad PR score.

Improve this answer
$\endgroup$
3
$\begingroup$

You're half way there.

Usually when I do imbalanced models, heck, even balanced models, I look at PR for ALL my classes.

In your example, yes, your positive class has P = 0.9 and R = 1.0. But what you should look at are ALL your classes. So for your negative class, your P = 0 and your R = 0. And you usually don't just look at PR scores individually. You want to look at F1-score (F1 macro or F1 micro, depending on your problem) that is a harmonic average of your PR scores for both class 1 and class 0. Your class 1 PR score is super good, but combine that with your class 0 PR score, your F1-score will be TERRIBLE, which is the correct conclusion for your scenario.

TL,DR: Look at PR scores for ALL your classes, and combine them with a metric like F1-score to have a realistic conclusion about your model performance. The F1-score for your scenario will be TERRIBLE, which is the correct conclusion for your scenario.

Improve this answer
$\endgroup$
0
$\begingroup$

Using ROC can miss the low precision where FP > TP because ROC only look at TPR and FPR. I think what we need to ask is:

  1. Will FP > TP happen with your data and the model.
  2. Does FP > TP matter for your business problem.

If the answer is yes for both, then ROC only is not fit for use.

enter image description here

If the problem is identifying shoplifters and FP will alarm the police. Almost all the shop customers are honest, then catching more honest customers as shoplifters than real ones will cause angry customers and police. Then looking at only ROC will not be a good idea to measure the model.

If the problem is identifying potentially fatal food for toddlers, FP > TP may not be a big issue because there will be so many safe food (TN) as long as TPR is really high. Then ROC will be fit for use to measure the model.

FP: False Positive
TP: True Positive
FPR: False Positive Rate
TPR: True Positive Rate

Suppose I am running a risky loan business where majority of the customers are risky. If I evaluate the business performance with ROC only, I think the performance is good because TPR is high and FPR is low, although actually the business is losing money because of FP > TP. By looking at PR, it will tell the low precision and I will understand the business performance is bad as I am looking money.

enter image description here

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.