ROC vs precision-and-recall curves

Question

I understand the formal differences between them, what I want to know is when it is more relevant to use one vs. the other.

• Do they always provide complementary insight about the performance of a given classification/detection system?
• When is it reasonable to provide them both, say, in a paper? instead of just one?
• Are there any alternative (maybe more modern) descriptors that capture the relevant aspects of both ROC and precision recall for a classification system?

I am interested in arguments for both binary and multi-class (e.g. as one-vs-all) cases.

Answer 1

The key difference is that ROC curves will be the same no matter what the baseline probability is, but PR curves may be more useful in practice for needle-in-haystack type problems or problems where the "positive" class is more interesting than the negative class.

To show this, first let's start with a very nice way to define precision, recall and specificity. Assume you have a "positive" class called 1 and a "negative" class called 0. $\hat{Y}$ is your estimate of the true class label $Y$. Then: $$ \begin{aligned} &\text{Precision} &= P(Y = 1 | \hat{Y} = 1) \\ &\text{Recall} = \text{Sensitivity} &= P(\hat{Y} = 1 | Y = 1) \\ &\text{Specificity} &= P(\hat{Y} = 0 | Y = 0) \end{aligned} $$ The key thing to note is that sensitivity/recall and specificity, which make up the ROC curve, are probabilities conditioned on the true class label. Therefore, they will be the same regardless of what $P(Y = 1)$ is. Precision is a probability conditioned on your estimate of the class label and will thus vary if you try your classifier in different populations with different baseline $P(Y = 1)$. However, it may be more useful in practice if you only care about one population with known background probability and the "positive" class is much more interesting than the "negative" class. (IIRC precision is popular in the document retrieval field, where this is the case.) This is because it directly answers the question, "What is the probability that this is a real hit given my classifier says it is?".

Interestingly, by Bayes' theorem you can work out cases where specificity can be very high and precision very low simultaneously. All you have to do is assume $P(Y = 1)$ is very close to zero. In practice I've developed several classifiers with this performance characteristic when searching for needles in DNA sequence haystacks.

IMHO when writing a paper you should provide whichever curve answers the question you want answered (or whichever one is more favorable to your method, if you're cynical). If your question is: "How meaningful is a positive result from my classifier given the baseline probabilities of my problem?", use a PR curve. If your question is, "How well can this classifier be expected to perform in general, at a variety of different baseline probabilities?", go with a ROC curve.

Answer 2

Here are the conclusions from a paper by Davis & Goadrich explaining the relationship between ROC and PR space. They answer the first two questions:

First, for any dataset, the ROC curve and PR curve for a given algorithm contain the same points. This equivalence, leads to the surprising theorem that a curve dominates in ROC space if and only if it dominates in PR space. Second, as a corollary to the theorem we show the existence of the PR space analog to the convex hull in ROC space, which we call achievable PR curve. Remarkably, when constructing the achievable PR curve one discards exactly the same points omit- ted by the convex hull in ROC space. Consequently, we can efficiently compute the achievable PR curve. [...] Finally, we show that an algorithm that optimizes the area under the ROC curve is not guaranteed to optimize the area under the PR curve.

In other words, in principle, ROC and PR are equally suited to compare results. But for the example case of a result of 20 hits and 1980 misses they show that the differences can be rather drastic, as shown in Figures 11 and 12.

Result/curve (I) describes a result where 10 of the 20 hits are in the top ten ranks and the remaining 10 hits are then evenly spread out over the first 1500 ranks. Resut (II) describes a result where the 20 hits are evenly spread over the first 500 (out of 2000) ranks. So in cases where a result "shape" like (I) is preferable, this preference is clearly distinguishable in PR-space, while the AUC ROC of the two results are nearly equal.

Answer 3

There is a lot of misunderstanding about evaluation. Part of this comes from the Machine Learning approach of trying to optimize algorithms on datasets, with no real interest in the data.

In a medical context, it's about the real world outcomes - how many people you save from dying, for example. In a medical context Sensitivity (TPR) is used to see how many of the positive cases are correctly picked up (minimizing the proportion missed as false negatives = FNR) while Specificity (TNR) is used to see how many of the negative cases are correctly eliminated (minimizing the proportion found as false positives = FPR). Some diseases have a prevalence of one in a million. Thus if you always predict negative you have an Accuracy of 0.999999 - this is achieved by the simple ZeroR learner that simply predicts the maximum class. If we consider the Recall and Precision for predicting that you are disease free, then we have Recall=1 and Precision=0.999999 for ZeroR. Of course, if you reverse +ve and -ve and try to predict that a person has the disease with ZeroR you get Recall=0 and Precision=undef (as you didn't even make a positive prediction, but often people define Precision as 0 in this case). Note that Recall (+ve Recall) and Inverse Recall (-ve Recall), and the related TPR,FPR,TNR & FNR are always defined because we are only tackling the problem because we know there are two classes to distinguish and we deliberately provide examples of each.

Note the huge difference between missing cancer in the medical context (someone dies and you get sued) versus missing a paper in a web search (good chance one of the others will reference it if its important). In both cases these errors are characterized as false negatives, versus a large population of negatives. In the websearch case we will automatically get a large population of true negatives simply because we only show a small number of results (e.g. 10 or 100) and not being shown shouldn't really be taking as a negative prediction (it might have been 101), whereas in the cancer test case we have a result for every person and unlike websearch we actively control the false negative level (rate).

So ROC is exploring the tradeoff between true positives (versus false negatives as a proportion of the real positives) and false positives (versus true negatives as a proportion of the real negatives). It is equivalent to comparing Sensitivity (+ve Recall) and Specificity (-ve Recall). There is also a PN graph which looks the same where we plot TP vs FP rather than TPR vs FPR - but since we make the plot square the only difference is the numbers we put on the scales. They are related by constants TPR=TP/RP, FPR=TP/RN where RP=TP+FN and RN=FN+FP are the number of Real Positives and Real Negatives in the dataset and conversely biases PP=TP+FP and PN=TN+FN are the number of times we Predict Positive or Predict Negative. Note that we call rp=RP/N and rn=RN/N the prevalence of positive resp. negative and pp=PP/N and rp=RP/N the bias to positive resp. negative.

If we sum or average Sensitivity and Specificity or look at the Area Under the tradeoff Curve (equivalent to ROC just reversing the x-axis) we get the same result if we interchange which class is +ve and +ve. This is NOT true for Precision and Recall (as illustrated above with disease prediction by ZeroR). This arbitrariness is a major deficiency of Precision, Recall and their averages (whether arithmetic, geometric or harmonic) and tradeoff graphs.

The PR, PN, ROC, LIFT and other charts are plotted as parameters of the system are changed. This classically plot points for each individual system trained, often with a threshold being increased or decreased to change the point at which an instance is classed positive versus negative.

Sometimes the plotted points may be averages over (changing parameters/thresholds/algorithms of) sets of systems trained in the same way (but using different random numbers or samplings or orderings). These are theoretical constructs that tell us about the average behaviour of the systems rather than their performance on a particular problem. The tradeoff charts are intended to help us choose the correct operating point for a particular application (dataset and approach) and this is where ROC gets its name from (Receiver Operating Characteristics aims to maximize the information received, in the sense of informedness).

Let us consider what Recall or TPR or TP can be plotted against.

TP vs FP (PN) - looks exactly like the ROC plot, just with different numbers

TPR vs FPR (ROC) - TPR against FPR with AUC is unchanged if +/- are reversed.

TPR vs TNR (alt ROC) - mirror image of ROC as TNR=1-FPR (TN+FP=RN)

TP vs PP (LIFT) - X incs for positive and negative examples (nonlinear stretch)

TPR vs pp (alt LIFT) - looks the same as LIFT, just with different numbers

TP vs 1/PP - very similar to LIFT (but inverted with nonlinear stretch)

TPR vs 1/PP - looks the same as TP vs 1/PP (different numbers on y-axis)

TP vs TP/PP - similar but with expansion of the x-axis (TP = X -> TP = X*TP)

TPR vs TP/PP - looks the same but with different numbers on the axes

The last is Recall vs Precision!

Note for these graphs any curves that dominate other curves (are better or at least as high at all points) will still dominate after these transformations. Since domination means "at least as high" at every point, the higher curve also has "at least as high" an Area under the Curve (AUC) as it includes also the area between the curves. The reverse is not true: if curves intersect, as opposed to touch, there is no dominance, but one AUC can still be bigger than the other.

All the transformations do is reflect and/or zoom in different (non-linear) ways to a particular part of the ROC or PN graph. However, only ROC has the nice interpretation of Area under the Curve (probability that a positive is ranked higher than a negative - Mann-Whitney U statistic) and Distance above the Curve (probability that an informed decision is made rather than guessing - Youden J statistic as the dichotomous form of Informedness).

Generally, there is no need to use the PR tradeoff curve and you can simply zoom into the ROC curve if detail is required. The ROC curve has the unique property that the diagonal (TPR=FPR) represents chance, that the Distance above the Chance line (DAC) represents Informedness or the probability of an informed decision, and the Area under the Curve (AUC) represents Rankedness or the probability of correct pairwise ranking. These results do not hold for the PR curve, and the AUC gets distorted for higher Recall or TPR as explained above. PR AUC being bigger does not imply ROC AUC is bigger and thus does not imply increased Rankedness (probability of ranked +/- pairs being correctly predicted - viz. how often it predicts +ves above -ves) and does not imply increased Informedness (probability of an informed prediction rather than a random guess - viz. how often it knows what it's doing when it makes a prediction).

Sorry - no graphs! If anyone wants to add graphs to illustrate the above transformations, that would be great! I do have quite a few in my papers about ROC, LIFT, BIRD, Kappa, F-measure, Informedness, etc. but they aren't presented in quite this way although there are illustrations of ROC vs LIFT vs BIRD vs RP in https://arxiv.org/pdf/1505.00401.pdf

UPDATE: To avoid trying to give full explanations in overlong answers or comments, here are some of my papers "discovering" the problem with Precision vs Recall tradeoffs inc. F1, deriving Informedness and then "exploring" the relationships with ROC, Kappa, Significance, DeltaP, AUC, etc. This is a problem one of my students bumped into 20 years ago (Entwisle) and many more have since found that realworld example of their own where there was empirical proof that the R/P/F/A approach sent the learner the WRONG way, while Informedness (or Kappa or Correlation in appropriate cases) sent them the RIGHT way - now across dozens of fields. There are also many good and relevant papers by other authors on Kappa and ROC, but when you use Kappas versus ROC AUC versus ROC Height (Informedness or Youden's J) is clarified in the 2012 papers I list (many of the important papers of others are cited in them). The 2003 Bookmaker paper derives for the first time a formula for Informedness for the multiclass case. The 2013 paper derives a multiclass version of Adaboost adapted to optimize Informedness (with links to the modified Weka that hosts and runs it).

References

1998 The present use of statistics in the evaluation of NLP parsers. J Entwisle, DMW Powers - Proceedings of the Joint Conferences on New Methods in Language Processing: 215-224 https://dl.acm.org/citation.cfm?id=1603935 Cited by 15

2003 Recall & Precision versus The Bookmaker. DMW Powers - International Conference on Cognitive Science: 529-534 http://dspace2.flinders.edu.au/xmlui/handle/2328/27159 Cited by 46

2011 Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. DMW Powers - Journal of Machine Learning Technology 2(1):37-63. http://dspace2.flinders.edu.au/xmlui/handle/2328/27165 Cited by 1749

2012 The problem with kappa. DMW Powers - Proceedings of the 13th Conference of the European ACL: 345-355 https://dl.acm.org/citation.cfm?id=2380859 Cited by 63

2012 ROC-ConCert: ROC-Based Measurement of Consistency and Certainty. DMW Powers - Spring Congress on Engineering and Technology (S-CET) 2:238-241 http://www.academia.edu/download/31939951/201203-SCET30795-ROC-ConCert-PID1124774.pdf Cited by 5

2013 ADABOOK & MULTIBOOK: : Adaptive Boosting with Chance Correction. DMW Powers- ICINCO International Conference on Informatics in Control, Automation and Robotics http://www.academia.edu/download/31947210/201309-AdaBook-ICINCO-SCITE-Harvard-2upcor_poster.pdf

https://www.dropbox.com/s/artzz1l3vozb6c4/weka.jar (goes into Java Class Path)
https://www.dropbox.com/s/dqws9ixew3egraj/wekagui   (GUI start script for Unix)
https://www.dropbox.com/s/4j3fwx997kq2xcq/wekagui.bat  (GUI shortcut on Windows)

Cited by 4

Answer 4

TL;DR

$AUC_{PvR}$ highlights the amount of False Positives relative to the class size, whereas $AUC_{ROC}$ better reflects the total amount of False Positives independent of in which class they come up.

Definition

The $AUC_{ROC}$ (receiver operator) is the area under the curve of true positive to false positive rate.

The $AUC_{PvR}$ (precision vs recall) is the area under the curve of the precison to recall metrics.

As true-positive rate equals recall, they only differ in comparing the recall to either precision or false-positive rate

$$ \begin{align}AUC_{PvR} =& AUC\left(\frac{\mathbf{TP}}{\mathbf{TP}+\mathbf{FP}}, \frac{TP}{TP + FN}\right)= AUC \left( \textbf{precision}, \text{recall}\right) \\AUC_{ROC} =& AUC\left(\frac{\mathbf{FP}}{\mathbf{FP} + \mathbf{TN}}, \frac{TP}{TP+FN}\right)= AUC \left(\textbf{FPR}, \text{recall} \right)\end{align} $$

Illustration

Consider an unbalanced multiclass dataset with a million instances. We are given two classifiers and observe their predictions at a fixed threshold for one of the classes:

Classifier A: 100 instances predicted (positive), 90 of which correctly (with 100 true cases)

Classifier B: 2000 instances predicted (positive), 90 of which correctly (with 100 true cases)

The respective values for the $AUC_{ROC}$ at the specific threshold will be

Classifier A: 0.9 TPR, 0.00001 FPR

Classifier: B: 0.9 TPR, 0.00191 FPR (gain of 0.0019)

The respective values for the $AUC_{PvR}$ for the specific threshold will be

Classifier A: 0.9 recall, 0.9 precision

Classifier B: 0.9 recall, 0.045 precision (gain of 0.855)

Discussion

As you can see, by choosing classifier B over A, the gain in false positive rate is comparably low compared to the gains observed in precision.

This is because the false-positive rate is a ratio of the false positives to the vast amount of true negatives, whereas the precision is a ratio of the false positives to the rather small amount of true positives.

Therefore the $AUC_{ROC}$ is more globally in that it is irritated by false positives relative to how many you could have drawn from the whole dataset. The $AUC_{PvR}$, however, is more local, in that it is irritated by false positives relative to how many positives you have set for the class at hand.

Your choice of metric, therefore, depends on what really irritates you more.

If you are irritated by a minority class having many false-positives relative to its class size $AUC_{PvR}$ will highlight this more thoroughly. (also false-positives of a majority class will have less impact)

If you are however more irritated by the global amount of false positives, the $AUC_{ROC}$ would be more indicative of how well you are doing.

Answer 5

I feel this arXiv paper from Jan/Feb 2024 should definitely be added to the discussion:

A Closer Look at AUROC and AUPRC under Class Imbalance

They discuss a few things, but mainly challenge the notion of always using AUPRC over AUROC in cases of class imbalance.

• They show both metrics can be rewritten in terms depending on FPR, implying that it might not always be best to use AUPRC blindly in cases of class imbalance
• They note a property AUPRC has where it gives a higher weight to samples that have higher scores, whereas AUROC weights all samples the same.

The second point implies a few things

(1) Due to ROC weighting all samples the same, regardless of the score the model gives them, it is ideal for model comparison

(2) Since PRC weights samples differently depending on their predicted score, there are issues when you have sub-populations (e.g., race, sex) that have different prevalence levels. This is an issue if you have fairness concerns

(3) However, PRC will be ideal if you want a model where a TP will be costly. So if you have a retrieval / top k problem and fairness isn't a concern, you should use AUPRC to optimize your model

(4) Let's say you are building a model for a rare disease (and thus imbalance in your training set), and you want your recall to be high, and you can then follow up with identified patients later. In this case AUPRC is actually damaging to your problem.

Note all the points in this paper pertain to determining which metric to use for training your model, not presenting results.

Overall I think it's an interesting paper presented nicely. Curious what takes other people have on this paper

Answer 6

Suppose I run a loan business. The total loan applications are sum of good loan applicants (AP) and bad applicants (AN). I find good applicants (TP) and give the loan but miss some good ones (FN). Likewise, I reject bad applicants (TN) but make mistakes and cause bad loan (FP).

The performance of my business will be measured with:

1. TPR to have correctly identified TP and maximized profit (gain).
2. 1-FPR to have correctly identified TN and minimized loss.
3. TP volume (profit) > FP volume (loss), otherwise I go bankrupt by debt excess.

ROC/AUC measures 1 and 2 which are ratios, however it does not tell 3. Hence I need 3 which is what Precision can tell.

Hence, in my opinion, we need both ROC/AUC and Precision/Recall to measure the model performance. However, under the condition that we know that AP and AN are more or less balanced, TPR/FPR can also indicate the Precision, hence we can omit Precision/Recall.

I suppose the reason why we wonder ROC or P/R is we may not be clearly aware of such conditions when we studied the Confusion Matrix, ROC, PR, and F1.

AP: Actual Positives = (TP + FN)
AN: Actual Negatives = (TN + FP)
TP: True Positive 
FP: False Positive
TN: True Negative
FN: False Negative
TPR: True Positive Rate = TP / (TP+FN)
FPR: False Positive Rate = FP / (FP + TN)
Precision = TP / (TP + FP)

Answer 7

Perhaps one helpful example to think about is information retrieval. Say you have 1,000,000 documents. 100 of them are relevant to a search query (i.e. positives), while the rest are irrelevant (i.e. negatives).

The important thing is that the "baseline" of retrieving the documents in a random order, with the 100 positive documents uniformly distributed through all 1,000,000 documents, is a very poor baseline which should be easily beaten by any search system. Typically, most of the irrelevant documents are obviously irrelevant and easily classified as such.

Let's say a basic search system ranks all 100 positive documents uniformly within the first 1000 documents, while a good search system ranks all 100 positive documents uniformly within the first 200 documents.

Now we see the problem with the ROC curve: for any search system, a recall (i.e. true positive rate) of 1 is reached for a very small false negative rate (before even 1% of negatives are misclassified as positive), and so the ROC curve (which plots recall against the false negative rate) almost immediately shoots up to 1.

However, the precision-recall (PR) curves will look very distinct. In this simple example, they will be flat lines of constant precision, but for the basic search system the precision will be 10%, while the good search system will have a precision of 50%.

Note that the x-axis for ROC effectively corresponds to all of the (very many) negative documents, while for PR the x-axis just corresponds to the positive documents. The lack of focus of ROC on the rare and interesting positive cases is what makes it less useful in this scenario.

In summary: in situations where the positive class is rare and we expect any decent classifier to rank most positives above almost all of the negatives, then PR is better at showing differences in performance than ROC.