3
$\begingroup$

Is it possible for a binary classifier to have a recall of 0.0 for one of the classes and at the same time an area under the ROC curve (AUC) of 1.0 for the same class?

$\endgroup$
  • $\begingroup$ I know a lot about classification but I have never heard the term recall. Could you give a definition? No practical classifier will have an AUC = 1. That amonts to perfect specification and perfect sensitivity. $\endgroup$ – Michael Chernick Sep 27 '12 at 19:56
  • $\begingroup$ Thank you @MichaelChernick for the info. In Recall–precision curves, recall = number of documents retrieved that are relevant / total number of documents that are relevant. ( / is divided by) $\endgroup$ – Jenn Sep 27 '12 at 20:04
  • $\begingroup$ @MichaelChernick TP/(TP+FN) or en.wikipedia.org/wiki/Recall_%28information_retrieval%29 $\endgroup$ – steffen Sep 27 '12 at 20:55
  • $\begingroup$ ROC is true positive rate(recall-pd) vs false positive rate (false alarm) graph. $\endgroup$ – user2892084 Feb 17 '14 at 9:59
  • $\begingroup$ -1. An AUC is not a classifier - it represents a collection of classifiers, differing by the cut-point of the predicted variable. As such, you can't have an AUC for the class - an AUC represents both classes. $\endgroup$ – Calimo Feb 27 '14 at 13:23
-3
$\begingroup$

ROC curves are false negative rate vs true positive rate graph. If you have AUC = 1, by definition you have perfect classifier.

From Information retrieval viewpoint ; if you have AUC = 1 then you have perfect recall and perfect precision. You recall all documents which exists about this topic, also all the documents you recall are relevant to your topic.

I would like to add more information for response to commenter.

Following is a graph from "ROC Graphs: Notes and Practical Considerations for Data Mining Researchers, Tom Fawcett"

Figure 2 of ROC Graphs

A discrete classifier is one that outputs only a class label. 
Each discrete classifier produces an (FP rate,TP rate) pair, 
which corresponds to a single point in ROC space. 
...
The point (0;1) represents perfect classification. 
D's performance is perfect as shown.
$\endgroup$
  • 6
    $\begingroup$ (-1) This answer is wrong: each ROC includes a point with recall 0 (trivial model always predicting the negative class). AUC 1 does not imply perfect recall and precision, it implies only that a dichtomization threshold exists which yields perfect recall (true positive rate; sensitivity) and fall-out (true negative rate; specificity). $\endgroup$ – cbeleites Feb 17 '14 at 15:20
  • 1
    $\begingroup$ @AtillaOzgur: by no means do I want to offend you, but I want to avoid that other readers with a similar question get off the track by the answer as it is. Please revise your answer, I'd like to reverse my downvote to an upvote! The practical conclusion on which I think we agree anyways is: something is wrong and the situation needs to be investigated more in depth. $\endgroup$ – cbeleites Feb 17 '14 at 20:16
  • 2
    $\begingroup$ @AtillaOzgur: yes, AUC = 1 is a perfect classifier. But see my (equally theoretical) exampe: even for that classifier the 0/0 and 1/1 points belong to the receiver operating curve. As your quote says: if you have an inherently dichotomous output, you do not get a receiver operating curve but a point in fpr/tpr space. How do you define the area under a curve if you don't have a curve but a single point? $\endgroup$ – cbeleites Feb 17 '14 at 22:25
  • 2
    $\begingroup$ Look at figure 4 of the paper you refer to: there you have the receiver operating curve of a perfect classifier with continuous output. The perfect classifier (in my sense 2) is the one obtained with the threshold that leads to 0 fpr / 1 tpr. But the area under the curve refers to all possible thresholds (meaning 1) and these include thresholds (e.g. the marked 0.5 and 0.6) where the working point is less than perfect. And it also includes the corners 0/0 and 1/1. Thus you can have a classifier (1) with AUC = 1 and choose a threshold that leads to a classifier (2) that has recall 0. $\endgroup$ – cbeleites Feb 17 '14 at 22:36
  • 1
    $\begingroup$ I second cbeleites. The key point, as he has stated, is "AUC 1 does not imply perfect recall and precision, it implies only that a dichtomization threshold exists which yields perfect recall", meanwhile this answer assumes that always the best threshold is selected. But as cbeleites has pointed out in his answer, this may not be the case. Nit-picky, but correct :) $\endgroup$ – steffen Feb 18 '14 at 10:45
10
$\begingroup$

Bit late to the party, but I think the chosen answer is wrong, so here's my reasoning:

I think the confusion comes from using the "a classifier" with two different meanings:

  • The ROC of "a classifier" considers a function that yields a continuous score as output which indicates the class. The ROC is generated by varying a threshold above which the label of the positive class is assigned.
  • On contrast, recall (or sensitivity or true positive rate TPR) implies that this threshold is fixed. So the recall of "a classifier" implies one further step: the dichotomization or hardening of the continuous output of "a classifier" in the ROC meaning.

The 2nd meaning classifier is one working point of a set of points (not entirely sure about term: the German term would be Schar) that together form the classifier in the first meaning, the parametrization being the threshold.

To generate the ROC, the threshold is varied from below the lowest observed score ("start") to above the highest observed score ("end"). This makes the curve start at (TPR 1.0; TNR 0.0) and end at (TPR 0.0; TNR 1.0). Note that these two points correspond to trivital classifiers (2nd meaning) that always predict the positive and negative class, respectively.

So each ROC contains at least one point (and "a classifier" in the 2nd sense) with recall 0.

Here's a graph produced from 4 cases with "predicted" scores 1, 2, 3, 4 and the first 2 cases and "reference" classes negative, negative, positive, positive. If you look at classifiers (2nd meaning) with threshold anywhere > 4, you end up at the red operating point which has recall 0, although the AUC is 1.0 for the classifier (1st meaning).

ROC

So far my example may look artificially constructed. After all, why would one in practice choose a threshold outside the predicted range?
However here are some points that come to my mind how one could be faced with such a situation in practice.

  • There are classifiers where the predicted scores actually have an interpretable meaning. Therefore, the threshold may be chosen according to some "external" principle. Consider a classifier like logistic regression which predicts class membership probability. Now consider a situation where you need to have rather specific predictions. For example, you decided that you'll raise the hurdles a bit and yell "positive" only if the predicted probability is above 90 %. Now if for some reason the predicted probabilities never reach the 90 % mark, you end up with the situation in question.

    The same would apply if the classifier was set up first as a chemical calibration, e.g. a quantitation of fasting blood glucose level. Prediction should be "diabetic" if the quantitation yields more than 126 mg/dl. Now assume that unfortunately, your recovery is bad: it predicts systematically far too low and 126 mg/dl is never predicted. A similar situation would result if you looked up the official threshold as 126 mg/dl, but calibrated mmol/l (126 mg/dl = 7,0 mmol/l)

  • Number two is a classical programming error. Consider programming the ROC by actually looping over a series of increasing threshold values to construct the ROC, e.g. setting some threshold variable accordigly in the loop. If then next step the recall is calculated using threshold which is still set to a value outside the predicted score range, the recall will be 0 (or 1, if high scores predict the negative class).

    ## calculate and plot the ROC
    roc <- data.frame (
             threshold = seq (min (score) - 1, max (score + 1), length.out = 100),
             tpr = NA, fpr = NA
           )
    
    for (i in 1 : 100) {
       threshold <- roc$threshold [i]
           roc$tpr [i] <- mean (score [class == "positive"] > threshold)
       roc$fpr [i] <- mean (score [class == "negative"] > threshold )
    }
    
    plot (roc$fpr, roc$tpr, col = rainbow (100))
    
    ## now the recall
    
    # **FORGET TO SET A SENSIBLE THRESHOLD VALUE**
    
    recall <-  mean (score [class == "positive"] > threshold)
    recall
    ## [1] 0
    

I used this R code to produce the graph:

require (ROCR)
perf <- performance (prediction (1:4, 1:4 > 2.5), "tpr", "fpr")
plot (perf, asp = 1)
points (perf@x.values [[1]], perf@y.values[[1]], pch = 19, col = c(2, 1,1,1,1))
$\endgroup$
  • $\begingroup$ Your red point is not AUC = 1, it is actually AUC = 0 point. AUC = 1 means that this point is D point as in my included graph. This point have to be in left uppermost $\endgroup$ – Atilla Ozgur Feb 17 '14 at 22:20
  • $\begingroup$ Note e.g. fig. 7 in the Fawcett 2004 report you linked: even for a classifiert that does not produce continuous scores but only two distinct values (the class labels coded e.g. as 0 and 1) the corresponding receiver operating curve which you need in order to calculate the area underneath extends to the 0/0 and 1/1 points. $\endgroup$ – cbeleites Feb 17 '14 at 22:48
  • $\begingroup$ AUC = 0 can only be obtained if the ROC goes through the lower right corner. $\endgroup$ – cbeleites Feb 17 '14 at 23:05
  • 3
    $\begingroup$ @AtillaOzgur the red point is not AUC = 0, nor AUC = 1. The AUC is defined by the ensemble of all red and black points. $\endgroup$ – Calimo Feb 27 '14 at 13:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.