162
$\begingroup$

I understand the formal differences between them, what I want to know is when it is more relevant to use one vs. the other.

  • Do they always provide complementary insight about the performance of a given classification/detection system?
  • When is it reasonable to provide them both, say, in a paper? instead of just one?
  • Are there any alternative (maybe more modern) descriptors that capture the relevant aspects of both ROC and precision recall for a classification system?

I am interested in arguments for both binary and multi-class (e.g. as one-vs-all) cases.

$\endgroup$
  • 7
    $\begingroup$ This paper just must appear in a context: biostat.wisc.edu/~page/rocpr.pdf $\endgroup$ – user88 Feb 14 '11 at 23:42
  • 2
    $\begingroup$ I might use this for a "plug" to mention my own thesis here... In Leitner (2012) I proposed an "F-measured Average Precision" (FAP) metric (see p. 65) as the harmonic mean of F-measure and Average Precision. I.e., a combination of a set evaluation metric with that of a ranked evaluation metric. In the thesis, I showed that maximizing the FAP score on the training set can be used to identify the best cutoff to delimit an otherwise unbounded information retrieval task (using 100s of BioCreative runs!). $\endgroup$ – fnl May 23 '17 at 13:15
  • 1
    $\begingroup$ Here is another good discussion on AUC-ROC and PR curve on an imbalanced dataset. It has the same conclusion as what dsimcha said. When you care more about the rare case, you should use PR. $\endgroup$ – Y.C. Feb 22 '18 at 3:38
215
$\begingroup$

The key difference is that ROC curves will be the same no matter what the baseline probability is, but PR curves may be more useful in practice for needle-in-haystack type problems or problems where the "positive" class is more interesting than the negative class.

To show this, first let's start with a very nice way to define precision, recall and specificity. Assume you have a "positive" class called 1 and a "negative" class called 0. $\hat{Y}$ is your estimate of the true class label $Y$. Then: $$ \begin{aligned} &\text{Precision} &= P(Y = 1 | \hat{Y} = 1) \\ &\text{Recall} = \text{Sensitivity} &= P(\hat{Y} = 1 | Y = 1) \\ &\text{Specificity} &= P(\hat{Y} = 0 | Y = 0) \end{aligned} $$ The key thing to note is that sensitivity/recall and specificity, which make up the ROC curve, are probabilities conditioned on the true class label. Therefore, they will be the same regardless of what $P(Y = 1)$ is. Precision is a probability conditioned on your estimate of the class label and will thus vary if you try your classifier in different populations with different baseline $P(Y = 1)$. However, it may be more useful in practice if you only care about one population with known background probability and the "positive" class is much more interesting than the "negative" class. (IIRC precision is popular in the document retrieval field, where this is the case.) This is because it directly answers the question, "What is the probability that this is a real hit given my classifier says it is?".

Interestingly, by Bayes' theorem you can work out cases where specificity can be very high and precision very low simultaneously. All you have to do is assume $P(Y = 1)$ is very close to zero. In practice I've developed several classifiers with this performance characteristic when searching for needles in DNA sequence haystacks.

IMHO when writing a paper you should provide whichever curve answers the question you want answered (or whichever one is more favorable to your method, if you're cynical). If your question is: "How meaningful is a positive result from my classifier given the baseline probabilities of my problem?", use a PR curve. If your question is, "How well can this classifier be expected to perform in general, at a variety of different baseline probabilities?", go with a ROC curve.

$\endgroup$
  • 10
    $\begingroup$ that was a fantastic explanation! $\endgroup$ – Amelio Vazquez-Reina Feb 14 '11 at 18:18
  • 5
    $\begingroup$ +1, great insight on the probabilistic interpretations of Precision, Recall and Specificity. $\endgroup$ – Zhubarb Jan 12 '15 at 14:26
  • 2
    $\begingroup$ What an answer ! Wish I could hit the up vote twice. $\endgroup$ – London guy Feb 12 '15 at 22:11
  • 6
    $\begingroup$ Just in case this was not clear from my earlier comment: This answer is wrong, as are ROC curves that use specificity. See, e.g., An Introduction to ROC analysis - which also hints at their shortcoming as documented in my answer: "Many real world domains are dominated by large numbers of negative instances, so performance in the far left-hand side of the ROC graph becomes more interesting." $\endgroup$ – fnl May 28 '16 at 13:37
  • 2
    $\begingroup$ +0.5 @fnl. While not explicitly wrong I think that the answer is missing the point of the question; the probabilistic interpretation is very welcome but it is moot in regards to the core question. In addition, I cannot come up with a generic realistic example where the question: "How meaningful is a positive result from my classifier given the baseline probabilities of my problem?" is inapplicable. The "in general" perspective of the ROC-AUC is just too fuzzy. (It goes without saying that neither should be used on face value to construct the final model) $\endgroup$ – usεr11852 says Reinstate Monic Apr 21 '17 at 18:39
27
$\begingroup$

Here are the conclusions from a paper by Davis & Goadrich explaining the relationship between ROC and PR space. They answer the first two questions:

First, for any dataset, the ROC curve and PR curve for a given algorithm contain the same points. This equivalence, leads to the surprising theorem that a curve dominates in ROC space if and only if it dominates in PR space. Second, as a corollary to the theorem we show the existence of the PR space analog to the convex hull in ROC space, which we call achievable PR curve. Remarkably, when constructing the achievable PR curve one discards exactly the same points omit- ted by the convex hull in ROC space. Consequently, we can efficiently compute the achievable PR curve. [...] Finally, we show that an algorithm that optimizes the area under the ROC curve is not guaranteed to optimize the area under the PR curve.

In other words, in principle, ROC and PR are equally suited to compare results. But for the example case of a result of 20 hits and 1980 misses they show that the differences can be rather drastic, as shown in Figures 11 and 12.

Figures 11 and 12 from Davis and Goadrich

Result/curve (I) describes a result where 10 of the 20 hits are in the top ten ranks and the remaining 10 hits are then evenly spread out over the first 1500 ranks. Resut (II) describes a result where the 20 hits are evenly spread over the first 500 (out of 2000) ranks. So in cases where a result "shape" like (I) is preferable, this preference is clearly distinguishable in PR-space, while the AUC ROC of the two results are nearly equal.

$\endgroup$
  • 1
    $\begingroup$ These graphs do not reflect (discretize) the situation described, which would show steps in the ROC curves every time a hit is encountered (after the first 10 for curve I). ROCCH would look like this with the Convex Hull. Similarly for PR, Precision would bump up a notch every time a hit was found, then decay during the misses, starting from (0,0) for nothing predicted (above threshold) if Precision was defined to be 0 at this point (0/0) - curve II as shown is the max Precision not the precision at each threshold (and hence Recall) level. $\endgroup$ – David M W Powers Nov 21 '17 at 14:05
  • 1
    $\begingroup$ This is actually Fig 7 in the version of the paper I found. The paper actually interpolates the PR curve using the ROC curve. Note that the domination result relies on the assumption that recall is nonzero, which is not the case until the first hit is found, and Precision (as defined in the paper) is formally undefined (0/0) until then. $\endgroup$ – David M W Powers Nov 21 '17 at 14:18
  • 1
    $\begingroup$ Yes, the lack of correct discretization is the issue (although a plot like this might occur if averaged over a large number of runs). However the paper's result is less meaningful than you might expect because of the undefinedness issues, and not as significant as you'd expect when you just understand the result in terms of rescaling. I would never use PR, but I would sometimes scale into ROC or equivalently use PN. $\endgroup$ – David M W Powers Nov 25 '17 at 1:58
  • 1
    $\begingroup$ First the graphs of Fig. 7 (11 vs 12) are irrelevant - they are not the stepped graphs for a trained system (as positive examples exceed a reducing threshold), but correspond to limit averages as the number of DIFFERENT systems approaches infinitity. Second Precision and Recall were desgined for web search and both totally IGNORE the (assumed large) number of true negatives (Prec=TP/PP and Rec=TP/RP). Third the Precision and Recall graph is really just showing the reciprocal bias (1/PP) vs reciprocal prevalence (1/RP) for a particular TP level (if you stopped a websearch at TP correct hits). $\endgroup$ – David M W Powers Nov 27 '17 at 7:00
  • 5
    $\begingroup$ OK, so after clearing all my doubts, I think it is necessary to advise readers that I believe @DavidMWPowers answer should be preferred over mine. $\endgroup$ – fnl Dec 1 '17 at 11:55
14
$\begingroup$

There is a lot of misunderstanding about evaluation. Part of this comes from the Machine Learning approach of trying to optimize algorithms on datasets, with no real interest in the data.

In a medical context, it's about the real world outcomes - how many people you save from dying, for example. In a medical context Sensitivity (TPR) is used to see how many of the positive cases are correctly picked up (minimizing the proportion missed as false negatives = FNR) while Specificity (TNR) is used to see how many of the negative cases are correctly eliminated (minimizing the proportion found as false positives = FPR). Some diseases have a prevalence of one in a million. Thus if you always predict negative you have an Accuracy of 0.999999 - this is achieved by the simple ZeroR learner that simply predicts the maximum class. If we consider the Recall and Precision for predicting that you are disease free, then we have Recall=1 and Precision=0.999999 for ZeroR. Of course, if you reverse +ve and -ve and try to predict that a person has the disease with ZeroR you get Recall=0 and Precision=undef (as you didn't even make a positive prediction, but often people define Precision as 0 in this case). Note that Recall (+ve Recall) and Inverse Recall (-ve Recall), and the related TPR,FPR,TNR & FNR are always defined because we are only tackling the problem because we know there are two classes to distinguish and we deliberately provide examples of each.

Note the huge difference between missing cancer in the medical context (someone dies and you get sued) versus missing a paper in a web search (good chance one of the others will reference it if its important). In both cases these errors are characterized as false negatives, versus a large population of negatives. In the websearch case we will automatically get a large population of true negatives simply because we only show a small number of results (e.g. 10 or 100) and not being shown shouldn't really be taking as a negative prediction (it might have been 101), whereas in the cancer test case we have a result for every person and unlike websearch we actively control the false negative level (rate).

So ROC is exploring the tradeoff between true positives (versus false negatives as a proportion of the real positives) and false positives (versus true negatives as a proportion of the real negatives). It is equivalent to comparing Sensitivity (+ve Recall) and Specificity (-ve Recall). There is also a PN graph which looks the same where we plot TP vs FP rather than TPR vs FPR - but since we make the plot square the only difference is the numbers we put on the scales. They are related by constants TPR=TP/RP, FPR=TP/RN where RP=TP+FN and RN=FN+FP are the number of Real Positives and Real Negatives in the dataset and conversely biases PP=TP+FP and PN=TN+FN are the number of times we Predict Positive or Predict Negative. Note that we call rp=RP/N and rn=RN/N the prevalence of positive resp. negative and pp=PP/N and rp=RP/N the bias to positive resp. negative.

If we sum or average Sensitivity and Specificity or look at the Area Under the tradeoff Curve (equivalent to ROC just reversing the x-axis) we get the same result if we interchange which class is +ve and +ve. This is NOT true for Precision and Recall (as illustrated above with disease prediction by ZeroR). This arbitrariness is a major deficiency of Precision, Recall and their averages (whether arithmetic, geometric or harmonic) and tradeoff graphs.

The PR, PN, ROC, LIFT and other charts are plotted as parameters of the system are changed. This classically plot points for each individual system trained, often with a threshold being increased or decreased to change the point at which an instance is classed positive versus negative.

Sometimes the plotted points may be averages over (changing parameters/thresholds/algorithms of) sets of systems trained in the same way (but using different random numbers or samplings or orderings). These are theoretical constructs that tell us about the average behaviour of the systems rather than their performance on a particular problem. The tradeoff charts are intended to help us choose the correct operating point for a particular application (dataset and approach) and this is where ROC gets its name from (Receiver Operating Characteristics aims to maximize the information received, in the sense of informedness).

Let us consider what Recall or TPR or TP can be plotted against.

TP vs FP (PN) - looks exactly like the ROC plot, just with different numbers

TPR vs FPR (ROC) - TPR against FPR with AUC is unchanged if +/- are reversed.

TPR vs TNR (alt ROC) - mirror image of ROC as TNR=1-FPR (TN+FP=RN)

TP vs PP (LIFT) - X incs for positive and negative examples (nonlinear stretch)

TPR vs pp (alt LIFT) - looks the same as LIFT, just with different numbers

TP vs 1/PP - very similar to LIFT (but inverted with nonlinear stretch)

TPR vs 1/PP - looks the same as TP vs 1/PP (different numbers on y-axis)

TP vs TP/PP - similar but with expansion of the x-axis (TP = X -> TP = X*TP)

TPR vs TP/PP - looks the same but with different numbers on the axes

The last is Recall vs Precision!

Note for these graphs any curves that dominate other curves (are better or at least as high at all points) will still dominate after these transformations. Since domination means "at least as high" at every point, the higher curve also has "at least as high" an Area under the Curve (AUC) as it includes also the area between the curves. The reverse is not true: if curves intersect, as opposed to touch, there is no dominance, but one AUC can still be bigger than the other.

All the transformations do is reflect and/or zoom in different (non-linear) ways to a particular part of the ROC or PN graph. However, only ROC has the nice interpretation of Area under the Curve (probability that a positive is ranked higher than a negative - Mann-Whitney U statistic) and Distance above the Curve (probability that an informed decision is made rather than guessing - Youden J statistic as the dichotomous form of Informedness).

Generally, there is no need to use the PR tradeoff curve and you can simply zoom into the ROC curve if detail is required. The ROC curve has the unique property that the diagonal (TPR=FPR) represents chance, that the Distance above the Chance line (DAC) represents Informedness or the probability of an informed decision, and the Area under the Curve (AUC) represents Rankedness or the probability of correct pairwise ranking. These results do not hold for the PR curve, and the AUC gets distorted for higher Recall or TPR as explained above. PR AUC being bigger does not imply ROC AUC is bigger and thus does not imply increased Rankedness (probability of ranked +/- pairs being correctly predicted - viz. how often it predicts +ves above -ves) and does not imply increased Informedness (probability of an informed prediction rather than a random guess - viz. how often it knows what it's doing when it makes a prediction).

Sorry - no graphs! If anyone wants to add graphs to illustrate the above transformations, that would be great! I do have quite a few in my papers about ROC, LIFT, BIRD, Kappa, F-measure, Informedness, etc. but they aren't presented in quite this way although there are illustrations of ROC vs LIFT vs BIRD vs RP in https://arxiv.org/pdf/1505.00401.pdf

UPDATE: To avoid trying to give full explanations in overlong answers or comments, here are some of my papers "discovering" the problem with Precision vs Recall tradeoffs inc. F1, deriving Informedness and then "exploring" the relationships with ROC, Kappa, Significance, DeltaP, AUC, etc. This is a problem one of my students bumped into 20 years ago (Entwisle) and many more have since found that realworld example of their own where there was empirical proof that the R/P/F/A approach sent the learner the WRONG way, while Informedness (or Kappa or Correlation in appropriate cases) sent them the RIGHT way - now across dozens of fields. There are also many good and relevant papers by other authors on Kappa and ROC, but when you use Kappas versus ROC AUC versus ROC Height (Informedness or Youden's J) is clarified in the 2012 papers I list (many of the important papers of others are cited in them). The 2003 Bookmaker paper derives for the first time a formula for Informedness for the multiclass case. The 2013 paper derives a multiclass version of Adaboost adapted to optimize Informedness (with links to the modified Weka that hosts and runs it).

References

1998 The present use of statistics in the evaluation of NLP parsers. J Entwisle, DMW Powers - Proceedings of the Joint Conferences on New Methods in Language Processing: 215-224 https://dl.acm.org/citation.cfm?id=1603935 Cited by 15

2003 Recall & Precision versus The Bookmaker. DMW Powers - International Conference on Cognitive Science: 529-534 http://dspace2.flinders.edu.au/xmlui/handle/2328/27159 Cited by 46

2011 Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. DMW Powers - Journal of Machine Learning Technology 2(1):37-63. http://dspace2.flinders.edu.au/xmlui/handle/2328/27165 Cited by 1749

2012 The problem with kappa. DMW Powers - Proceedings of the 13th Conference of the European ACL: 345-355 https://dl.acm.org/citation.cfm?id=2380859 Cited by 63

2012 ROC-ConCert: ROC-Based Measurement of Consistency and Certainty. DMW Powers - Spring Congress on Engineering and Technology (S-CET) 2:238-241 http://www.academia.edu/download/31939951/201203-SCET30795-ROC-ConCert-PID1124774.pdf Cited by 5

2013 ADABOOK & MULTIBOOK: : Adaptive Boosting with Chance Correction. DMW Powers- ICINCO International Conference on Informatics in Control, Automation and Robotics http://www.academia.edu/download/31947210/201309-AdaBook-ICINCO-SCITE-Harvard-2upcor_poster.pdf

https://www.dropbox.com/s/artzz1l3vozb6c4/weka.jar (goes into Java Class Path)
https://www.dropbox.com/s/dqws9ixew3egraj/wekagui   (GUI start script for Unix)
https://www.dropbox.com/s/4j3fwx997kq2xcq/wekagui.bat  (GUI shortcut on Windows)

Cited by 4

$\endgroup$
  • $\begingroup$ > "the area under the curve represents Rankedness or the probability of correct pairwise ranking" I guess, that is exactly where we disagree - the ROC only demonstrates ranking quality in the plot. However, with the AUC PR is a single number that immediately tells me if which ranking is preferable (i.e., that result I is preferable over result II). The AUC ROC does not have this property. $\endgroup$ – fnl Nov 27 '17 at 11:10
  • $\begingroup$ The domination result fnl cites means that WHEN DEFINED, If one curve dominates in ROC it does in PR and vice-versa, and this means it also has higher area in both, and thus there is no qualitative difference between ROC and PR AUC. The quote about Rankedness (Mann-Whitney U) is a well established quantitative result re probability (part of a significance test) that was recommended quite independently of ROC, but was later found to be ROC AUC. Similarly Informedness was originally defined independently, and later proven to correspond to height of ROC's operating point. PR has no such results. $\endgroup$ – David M W Powers Nov 27 '17 at 23:21
  • 1
    $\begingroup$ As I've said before this is just a matter of scaling under domination conditions ("much larger" because multiplied by a big number as I explain in detail), but under non-domination conditions AUC PR is misleading and AUC ROC is the one that has an appropriate probabilistic interpretation (Mann-Whitney U or Rankedness), with the single operating point case corresponding to Gini (or equivalently Youden's J or Informedness, after scaling). $\endgroup$ – David M W Powers Nov 29 '17 at 10:40
  • 1
    $\begingroup$ If we consider the single operating point (SOC) AUC for simplicity, then Gini Coefficient = AUC = (TP/RP + TN/RN)/2 and Informedness = Youden J = TP/RP + TN/RN - 1 = Sensitivity + Specificity -1 = TPR + TNF -1 = Recall + Inverse Recall - 1 etc. Maximizing either is equivalent, but the latter is the probability of an informed decision (deliberately the wrong one if -ve). If RN and TN both go to infinity with TN>>FP then TN/RN -> 1 and cancels so Informedness = Recall in the cases you cite. If instead the huge class is RP and TP>>FN then TP/RP -> 1 and Informedness = Inverse Recall. See refs. $\endgroup$ – David M W Powers Nov 30 '17 at 12:03
  • 1
    $\begingroup$ This is a very helpful answer David Powers. But forgive my ignorance, when you say, 'Generally, there is no need to use the PR tradeoff curve and you can simply zoom into the ROC curve if detail is required.', how exactly do I do that and could you give any more detail about what you mean? Does this mean I can use a ROC curve in a severely imbalanced case somehow? 'Giving the FPR or TPR a higher weight would produce an AUC ROC score with larger result differences, excellent point!' How do I do this then with my ROC? $\endgroup$ – Christopher John Sep 27 at 12:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.