# ROC vs precision-and-recall curves

I understand the formal differences between them, what I want to know is when it is more relevant to use one vs. the other.

• Do they always provide complementary insight about the performance of a given classification/detection system?
• When is it reasonable to provide them both, say, in a paper? instead of just one?
• Are there any alternative (maybe more modern) descriptors that capture the relevant aspects of both ROC and precision recall for a classification system?

I am interested in arguments for both binary and multi-class (e.g. as one-vs-all) cases.

• This paper just must appear in a context: biostat.wisc.edu/~page/rocpr.pdf
– user88
Feb 14, 2011 at 23:42
• I might use this for a "plug" to mention my own thesis here... In Leitner (2012) I proposed an "F-measured Average Precision" (FAP) metric (see p. 65) as the harmonic mean of F-measure and Average Precision. I.e., a combination of a set evaluation metric with that of a ranked evaluation metric. In the thesis, I showed that maximizing the FAP score on the training set can be used to identify the best cutoff to delimit an otherwise unbounded information retrieval task (using 100s of BioCreative runs!).
– fnl
May 23, 2017 at 13:15
• Here is another good discussion on AUC-ROC and PR curve on an imbalanced dataset. It has the same conclusion as what dsimcha said. When you care more about the rare case, you should use PR. Feb 22, 2018 at 3:38

The key difference is that ROC curves will be the same no matter what the baseline probability is, but PR curves may be more useful in practice for needle-in-haystack type problems or problems where the "positive" class is more interesting than the negative class.

To show this, first let's start with a very nice way to define precision, recall and specificity. Assume you have a "positive" class called 1 and a "negative" class called 0. $\hat{Y}$ is your estimate of the true class label $Y$. Then: \begin{aligned} &\text{Precision} &= P(Y = 1 | \hat{Y} = 1) \\ &\text{Recall} = \text{Sensitivity} &= P(\hat{Y} = 1 | Y = 1) \\ &\text{Specificity} &= P(\hat{Y} = 0 | Y = 0) \end{aligned} The key thing to note is that sensitivity/recall and specificity, which make up the ROC curve, are probabilities conditioned on the true class label. Therefore, they will be the same regardless of what $P(Y = 1)$ is. Precision is a probability conditioned on your estimate of the class label and will thus vary if you try your classifier in different populations with different baseline $P(Y = 1)$. However, it may be more useful in practice if you only care about one population with known background probability and the "positive" class is much more interesting than the "negative" class. (IIRC precision is popular in the document retrieval field, where this is the case.) This is because it directly answers the question, "What is the probability that this is a real hit given my classifier says it is?".

Interestingly, by Bayes' theorem you can work out cases where specificity can be very high and precision very low simultaneously. All you have to do is assume $P(Y = 1)$ is very close to zero. In practice I've developed several classifiers with this performance characteristic when searching for needles in DNA sequence haystacks.

IMHO when writing a paper you should provide whichever curve answers the question you want answered (or whichever one is more favorable to your method, if you're cynical). If your question is: "How meaningful is a positive result from my classifier given the baseline probabilities of my problem?", use a PR curve. If your question is, "How well can this classifier be expected to perform in general, at a variety of different baseline probabilities?", go with a ROC curve.

• that was a fantastic explanation! Feb 14, 2011 at 18:18
• +1, great insight on the probabilistic interpretations of Precision, Recall and Specificity. Jan 12, 2015 at 14:26
• What an answer ! Wish I could hit the up vote twice. Feb 12, 2015 at 22:11
• Just in case this was not clear from my earlier comment: This answer is wrong, as are ROC curves that use specificity. See, e.g., An Introduction to ROC analysis - which also hints at their shortcoming as documented in my answer: "Many real world domains are dominated by large numbers of negative instances, so performance in the far left-hand side of the ROC graph becomes more interesting."
– fnl
May 28, 2016 at 13:37
• +0.5 @fnl. While not explicitly wrong I think that the answer is missing the point of the question; the probabilistic interpretation is very welcome but it is moot in regards to the core question. In addition, I cannot come up with a generic realistic example where the question: "How meaningful is a positive result from my classifier given the baseline probabilities of my problem?" is inapplicable. The "in general" perspective of the ROC-AUC is just too fuzzy. (It goes without saying that neither should be used on face value to construct the final model) Apr 21, 2017 at 18:39

Here are the conclusions from a paper by Davis & Goadrich explaining the relationship between ROC and PR space. They answer the first two questions:

First, for any dataset, the ROC curve and PR curve for a given algorithm contain the same points. This equivalence, leads to the surprising theorem that a curve dominates in ROC space if and only if it dominates in PR space. Second, as a corollary to the theorem we show the existence of the PR space analog to the convex hull in ROC space, which we call achievable PR curve. Remarkably, when constructing the achievable PR curve one discards exactly the same points omit- ted by the convex hull in ROC space. Consequently, we can efficiently compute the achievable PR curve. [...] Finally, we show that an algorithm that optimizes the area under the ROC curve is not guaranteed to optimize the area under the PR curve.

In other words, in principle, ROC and PR are equally suited to compare results. But for the example case of a result of 20 hits and 1980 misses they show that the differences can be rather drastic, as shown in Figures 11 and 12. Result/curve (I) describes a result where 10 of the 20 hits are in the top ten ranks and the remaining 10 hits are then evenly spread out over the first 1500 ranks. Resut (II) describes a result where the 20 hits are evenly spread over the first 500 (out of 2000) ranks. So in cases where a result "shape" like (I) is preferable, this preference is clearly distinguishable in PR-space, while the AUC ROC of the two results are nearly equal.

• These graphs do not reflect (discretize) the situation described, which would show steps in the ROC curves every time a hit is encountered (after the first 10 for curve I). ROCCH would look like this with the Convex Hull. Similarly for PR, Precision would bump up a notch every time a hit was found, then decay during the misses, starting from (0,0) for nothing predicted (above threshold) if Precision was defined to be 0 at this point (0/0) - curve II as shown is the max Precision not the precision at each threshold (and hence Recall) level. Nov 21, 2017 at 14:05
• This is actually Fig 7 in the version of the paper I found. The paper actually interpolates the PR curve using the ROC curve. Note that the domination result relies on the assumption that recall is nonzero, which is not the case until the first hit is found, and Precision (as defined in the paper) is formally undefined (0/0) until then. Nov 21, 2017 at 14:18
• Yes, the lack of correct discretization is the issue (although a plot like this might occur if averaged over a large number of runs). However the paper's result is less meaningful than you might expect because of the undefinedness issues, and not as significant as you'd expect when you just understand the result in terms of rescaling. I would never use PR, but I would sometimes scale into ROC or equivalently use PN. Nov 25, 2017 at 1:58
• First the graphs of Fig. 7 (11 vs 12) are irrelevant - they are not the stepped graphs for a trained system (as positive examples exceed a reducing threshold), but correspond to limit averages as the number of DIFFERENT systems approaches infinitity. Second Precision and Recall were desgined for web search and both totally IGNORE the (assumed large) number of true negatives (Prec=TP/PP and Rec=TP/RP). Third the Precision and Recall graph is really just showing the reciprocal bias (1/PP) vs reciprocal prevalence (1/RP) for a particular TP level (if you stopped a websearch at TP correct hits). Nov 27, 2017 at 7:00
• OK, so after clearing all my doubts, I think it is necessary to advise readers that I believe @DavidMWPowers answer should be preferred over mine.
– fnl
Dec 1, 2017 at 11:55

There is a lot of misunderstanding about evaluation. Part of this comes from the Machine Learning approach of trying to optimize algorithms on datasets, with no real interest in the data.

In a medical context, it's about the real world outcomes - how many people you save from dying, for example. In a medical context Sensitivity (TPR) is used to see how many of the positive cases are correctly picked up (minimizing the proportion missed as false negatives = FNR) while Specificity (TNR) is used to see how many of the negative cases are correctly eliminated (minimizing the proportion found as false positives = FPR). Some diseases have a prevalence of one in a million. Thus if you always predict negative you have an Accuracy of 0.999999 - this is achieved by the simple ZeroR learner that simply predicts the maximum class. If we consider the Recall and Precision for predicting that you are disease free, then we have Recall=1 and Precision=0.999999 for ZeroR. Of course, if you reverse +ve and -ve and try to predict that a person has the disease with ZeroR you get Recall=0 and Precision=undef (as you didn't even make a positive prediction, but often people define Precision as 0 in this case). Note that Recall (+ve Recall) and Inverse Recall (-ve Recall), and the related TPR,FPR,TNR & FNR are always defined because we are only tackling the problem because we know there are two classes to distinguish and we deliberately provide examples of each.

Note the huge difference between missing cancer in the medical context (someone dies and you get sued) versus missing a paper in a web search (good chance one of the others will reference it if its important). In both cases these errors are characterized as false negatives, versus a large population of negatives. In the websearch case we will automatically get a large population of true negatives simply because we only show a small number of results (e.g. 10 or 100) and not being shown shouldn't really be taking as a negative prediction (it might have been 101), whereas in the cancer test case we have a result for every person and unlike websearch we actively control the false negative level (rate).

So ROC is exploring the tradeoff between true positives (versus false negatives as a proportion of the real positives) and false positives (versus true negatives as a proportion of the real negatives). It is equivalent to comparing Sensitivity (+ve Recall) and Specificity (-ve Recall). There is also a PN graph which looks the same where we plot TP vs FP rather than TPR vs FPR - but since we make the plot square the only difference is the numbers we put on the scales. They are related by constants TPR=TP/RP, FPR=TP/RN where RP=TP+FN and RN=FN+FP are the number of Real Positives and Real Negatives in the dataset and conversely biases PP=TP+FP and PN=TN+FN are the number of times we Predict Positive or Predict Negative. Note that we call rp=RP/N and rn=RN/N the prevalence of positive resp. negative and pp=PP/N and rp=RP/N the bias to positive resp. negative.

If we sum or average Sensitivity and Specificity or look at the Area Under the tradeoff Curve (equivalent to ROC just reversing the x-axis) we get the same result if we interchange which class is +ve and +ve. This is NOT true for Precision and Recall (as illustrated above with disease prediction by ZeroR). This arbitrariness is a major deficiency of Precision, Recall and their averages (whether arithmetic, geometric or harmonic) and tradeoff graphs.

The PR, PN, ROC, LIFT and other charts are plotted as parameters of the system are changed. This classically plot points for each individual system trained, often with a threshold being increased or decreased to change the point at which an instance is classed positive versus negative.

Sometimes the plotted points may be averages over (changing parameters/thresholds/algorithms of) sets of systems trained in the same way (but using different random numbers or samplings or orderings). These are theoretical constructs that tell us about the average behaviour of the systems rather than their performance on a particular problem. The tradeoff charts are intended to help us choose the correct operating point for a particular application (dataset and approach) and this is where ROC gets its name from (Receiver Operating Characteristics aims to maximize the information received, in the sense of informedness).

Let us consider what Recall or TPR or TP can be plotted against.

TP vs FP (PN) - looks exactly like the ROC plot, just with different numbers

TPR vs FPR (ROC) - TPR against FPR with AUC is unchanged if +/- are reversed.

TPR vs TNR (alt ROC) - mirror image of ROC as TNR=1-FPR (TN+FP=RN)

TP vs PP (LIFT) - X incs for positive and negative examples (nonlinear stretch)

TPR vs pp (alt LIFT) - looks the same as LIFT, just with different numbers

TP vs 1/PP - very similar to LIFT (but inverted with nonlinear stretch)

TPR vs 1/PP - looks the same as TP vs 1/PP (different numbers on y-axis)

TP vs TP/PP - similar but with expansion of the x-axis (TP = X -> TP = X*TP)

TPR vs TP/PP - looks the same but with different numbers on the axes

The last is Recall vs Precision!

Note for these graphs any curves that dominate other curves (are better or at least as high at all points) will still dominate after these transformations. Since domination means "at least as high" at every point, the higher curve also has "at least as high" an Area under the Curve (AUC) as it includes also the area between the curves. The reverse is not true: if curves intersect, as opposed to touch, there is no dominance, but one AUC can still be bigger than the other.

All the transformations do is reflect and/or zoom in different (non-linear) ways to a particular part of the ROC or PN graph. However, only ROC has the nice interpretation of Area under the Curve (probability that a positive is ranked higher than a negative - Mann-Whitney U statistic) and Distance above the Curve (probability that an informed decision is made rather than guessing - Youden J statistic as the dichotomous form of Informedness).

Generally, there is no need to use the PR tradeoff curve and you can simply zoom into the ROC curve if detail is required. The ROC curve has the unique property that the diagonal (TPR=FPR) represents chance, that the Distance above the Chance line (DAC) represents Informedness or the probability of an informed decision, and the Area under the Curve (AUC) represents Rankedness or the probability of correct pairwise ranking. These results do not hold for the PR curve, and the AUC gets distorted for higher Recall or TPR as explained above. PR AUC being bigger does not imply ROC AUC is bigger and thus does not imply increased Rankedness (probability of ranked +/- pairs being correctly predicted - viz. how often it predicts +ves above -ves) and does not imply increased Informedness (probability of an informed prediction rather than a random guess - viz. how often it knows what it's doing when it makes a prediction).

Sorry - no graphs! If anyone wants to add graphs to illustrate the above transformations, that would be great! I do have quite a few in my papers about ROC, LIFT, BIRD, Kappa, F-measure, Informedness, etc. but they aren't presented in quite this way although there are illustrations of ROC vs LIFT vs BIRD vs RP in https://arxiv.org/pdf/1505.00401.pdf

UPDATE: To avoid trying to give full explanations in overlong answers or comments, here are some of my papers "discovering" the problem with Precision vs Recall tradeoffs inc. F1, deriving Informedness and then "exploring" the relationships with ROC, Kappa, Significance, DeltaP, AUC, etc. This is a problem one of my students bumped into 20 years ago (Entwisle) and many more have since found that realworld example of their own where there was empirical proof that the R/P/F/A approach sent the learner the WRONG way, while Informedness (or Kappa or Correlation in appropriate cases) sent them the RIGHT way - now across dozens of fields. There are also many good and relevant papers by other authors on Kappa and ROC, but when you use Kappas versus ROC AUC versus ROC Height (Informedness or Youden's J) is clarified in the 2012 papers I list (many of the important papers of others are cited in them). The 2003 Bookmaker paper derives for the first time a formula for Informedness for the multiclass case. The 2013 paper derives a multiclass version of Adaboost adapted to optimize Informedness (with links to the modified Weka that hosts and runs it).

References

1998 The present use of statistics in the evaluation of NLP parsers. J Entwisle, DMW Powers - Proceedings of the Joint Conferences on New Methods in Language Processing: 215-224 https://dl.acm.org/citation.cfm?id=1603935 Cited by 15

2003 Recall & Precision versus The Bookmaker. DMW Powers - International Conference on Cognitive Science: 529-534 http://dspace2.flinders.edu.au/xmlui/handle/2328/27159 Cited by 46

2011 Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. DMW Powers - Journal of Machine Learning Technology 2(1):37-63. http://dspace2.flinders.edu.au/xmlui/handle/2328/27165 Cited by 1749

2012 The problem with kappa. DMW Powers - Proceedings of the 13th Conference of the European ACL: 345-355 https://dl.acm.org/citation.cfm?id=2380859 Cited by 63

2012 ROC-ConCert: ROC-Based Measurement of Consistency and Certainty. DMW Powers - Spring Congress on Engineering and Technology (S-CET) 2:238-241 http://www.academia.edu/download/31939951/201203-SCET30795-ROC-ConCert-PID1124774.pdf Cited by 5

2013 ADABOOK & MULTIBOOK: : Adaptive Boosting with Chance Correction. DMW Powers- ICINCO International Conference on Informatics in Control, Automation and Robotics http://www.academia.edu/download/31947210/201309-AdaBook-ICINCO-SCITE-Harvard-2upcor_poster.pdf

https://www.dropbox.com/s/artzz1l3vozb6c4/weka.jar (goes into Java Class Path)
https://www.dropbox.com/s/dqws9ixew3egraj/wekagui   (GUI start script for Unix)
https://www.dropbox.com/s/4j3fwx997kq2xcq/wekagui.bat  (GUI shortcut on Windows)


Cited by 4

• > "the area under the curve represents Rankedness or the probability of correct pairwise ranking" I guess, that is exactly where we disagree - the ROC only demonstrates ranking quality in the plot. However, with the AUC PR is a single number that immediately tells me if which ranking is preferable (i.e., that result I is preferable over result II). The AUC ROC does not have this property.
– fnl
Nov 27, 2017 at 11:10
• As I've said before this is just a matter of scaling under domination conditions ("much larger" because multiplied by a big number as I explain in detail), but under non-domination conditions AUC PR is misleading and AUC ROC is the one that has an appropriate probabilistic interpretation (Mann-Whitney U or Rankedness), with the single operating point case corresponding to Gini (or equivalently Youden's J or Informedness, after scaling). Nov 29, 2017 at 10:40
• If we consider the single operating point (SOC) AUC for simplicity, then Gini Coefficient = AUC = (TP/RP + TN/RN)/2 and Informedness = Youden J = TP/RP + TN/RN - 1 = Sensitivity + Specificity -1 = TPR + TNF -1 = Recall + Inverse Recall - 1 etc. Maximizing either is equivalent, but the latter is the probability of an informed decision (deliberately the wrong one if -ve). If RN and TN both go to infinity with TN>>FP then TN/RN -> 1 and cancels so Informedness = Recall in the cases you cite. If instead the huge class is RP and TP>>FN then TP/RP -> 1 and Informedness = Inverse Recall. See refs. Nov 30, 2017 at 12:03
• This is a very helpful answer David Powers. But forgive my ignorance, when you say, 'Generally, there is no need to use the PR tradeoff curve and you can simply zoom into the ROC curve if detail is required.', how exactly do I do that and could you give any more detail about what you mean? Does this mean I can use a ROC curve in a severely imbalanced case somehow? 'Giving the FPR or TPR a higher weight would produce an AUC ROC score with larger result differences, excellent point!' How do I do this then with my ROC? Sep 27, 2019 at 12:45
• @Christopher John. Zooming in to the ROC curve or scaling the ROC curve can be done by changing the axes. Usually you plot 0..1 domain and range, but if you have a very large negative class RN relative to the true and predicted positives RP and RN, then FPR will tend to be small. So for example you could just limit the FPR axis to say 0..0.1 for a 10:1 ratio or 0..0.01 for 1 100:1 ratio. Similarly if TPR is close to 1 you could look at 0.9..1 or 0.99..1 for corresponding ratios (viz. FNR of 0..0.1 or 0..0.01 since TPR+FNR=TNR+FPR=1) Dec 30, 2019 at 0:58

# TL;DR

$$AUC_{PvR}$$ highlights the amount of False Positives relative to the class size, whereas $$AUC_{ROC}$$ better reflects the total amount of False Positives independent of in which class they come up.

# Definition

The $$AUC_{ROC}$$ (receiver operator) is the area under the curve of true positive to false positive rate.

The $$AUC_{PvR}$$ (precision vs recall) is the area under the curve of the precison to recall metrics.

As true-positive rate equals recall, they only differ in comparing the recall to either precision or false-positive rate

\begin{align}AUC_{PvR} =& AUC\left(\frac{\mathbf{TP}}{\mathbf{TP}+\mathbf{FP}}, \frac{TP}{TP + FN}\right)= AUC \left( \textbf{precision}, \text{recall}\right) \\AUC_{ROC} =& AUC\left(\frac{\mathbf{FP}}{\mathbf{FP} + \mathbf{TN}}, \frac{TP}{TP+FN}\right)= AUC \left(\textbf{FPR}, \text{recall} \right)\end{align}

# Illustration

Consider an unbalanced multiclass dataset with a million instances. We are given two classifiers and observe their predictions at a fixed threshold for one of the classes:

Classifier A: 100 instances predicted (positive), 90 of which correctly (with 100 true cases)

Classifier B: 2000 instances predicted (positive), 90 of which correctly (with 100 true cases)

The respective values for the $$AUC_{ROC}$$ at the specific threshold will be

Classifier A: 0.9 TPR, 0.00001 FPR

Classifier: B: 0.9 TPR, 0.00191 FPR (gain of 0.0019)

The respective values for the $$AUC_{PvR}$$ for the specific threshold will be

Classifier A: 0.9 recall, 0.9 precision

Classifier B: 0.9 recall, 0.045 precision (gain of 0.855)

# Discussion

As you can see, by choosing classifier B over A, the gain in false positive rate is comparably low compared to the gains observed in precision.

This is because the false-positive rate is a ratio of the false positives to the vast amount of true negatives, whereas the precision is a ratio of the false positives to the rather small amount of true positives.

Therefore the $$AUC_{ROC}$$ is more globally in that it is irritated by false positives relative to how many you could have drawn from the whole dataset. The $$AUC_{PvR}$$, however, is more local, in that it is irritated by false positives relative to how many positives you have set for the class at hand.

Your choice of metric, therefore, depends on what really irritates you more.

If you are irritated by a minority class having many false-positives relative to its class size $$AUC_{PvR}$$ will highlight this more thoroughly. (also false-positives of a majority class will have less impact)

If you are however more irritated by the global amount of false positives, the $$AUC_{ROC}$$ would be more indicative of how well you are doing.