Recall and AUC of a binary classifier

Question

Is it possible for a binary classifier to have a recall of 0.0 for one of the classes and at the same time an area under the ROC curve (AUC) of 1.0 for the same class?

Answer 1

ROC curves are false negative rate vs true positive rate graph. If you have AUC = 1, by definition you have perfect classifier.

From Information retrieval viewpoint ; if you have AUC = 1 then you have perfect recall and perfect precision. You recall all documents which exists about this topic, also all the documents you recall are relevant to your topic.

I would like to add more information for response to commenter.

Following is a graph from "ROC Graphs: Notes and Practical Considerations for Data Mining Researchers, Tom Fawcett"

A discrete classifier is one that outputs only a class label. 
Each discrete classifier produces an (FP rate,TP rate) pair, 
which corresponds to a single point in ROC space. 
...
The point (0;1) represents perfect classification. 
D's performance is perfect as shown.

Answer 2

Bit late to the party, but I think the chosen answer is wrong, so here's my reasoning:

I think the confusion comes from using the "a classifier" with two different meanings:

• The ROC of "a classifier" considers a function that yields a continuous score as output which indicates the class. The ROC is generated by varying a threshold above which the label of the positive class is assigned.
• On contrast, recall (or sensitivity or true positive rate TPR) implies that this threshold is fixed. So the recall of "a classifier" implies one further step: the dichotomization or hardening of the continuous output of "a classifier" in the ROC meaning.

The 2nd meaning classifier is one working point of a set of points (not entirely sure about term: the German term would be Schar) that together form the classifier in the first meaning, the parametrization being the threshold.

To generate the ROC, the threshold is varied from below the lowest observed score ("start") to above the highest observed score ("end"). This makes the curve start at (TPR 1.0; TNR 0.0) and end at (TPR 0.0; TNR 1.0). Note that these two points correspond to trivital classifiers (2nd meaning) that always predict the positive and negative class, respectively.

So each ROC contains at least one point (and "a classifier" in the 2nd sense) with recall 0.

Here's a graph produced from 4 cases with "predicted" scores 1, 2, 3, 4 and the first 2 cases and "reference" classes negative, negative, positive, positive. If you look at classifiers (2nd meaning) with threshold anywhere > 4, you end up at the red operating point which has recall 0, although the AUC is 1.0 for the classifier (1st meaning).

So far my example may look artificially constructed. After all, why would one in practice choose a threshold outside the predicted range?
However here are some points that come to my mind how one could be faced with such a situation in practice.

• There are classifiers where the predicted scores actually have an interpretable meaning. Therefore, the threshold may be chosen according to some "external" principle. Consider a classifier like logistic regression which predicts class membership probability. Now consider a situation where you need to have rather specific predictions. For example, you decided that you'll raise the hurdles a bit and yell "positive" only if the predicted probability is above 90 %. Now if for some reason the predicted probabilities never reach the 90 % mark, you end up with the situation in question.

  The same would apply if the classifier was set up first as a chemical calibration, e.g. a quantitation of fasting blood glucose level. Prediction should be "diabetic" if the quantitation yields more than 126 mg/dl. Now assume that unfortunately, your recovery is bad: it predicts systematically far too low and 126 mg/dl is never predicted. A similar situation would result if you looked up the official threshold as 126 mg/dl, but calibrated mmol/l (126 mg/dl = 7,0 mmol/l)

• Number two is a classical programming error. Consider programming the ROC by actually looping over a series of increasing threshold values to construct the ROC, e.g. setting some threshold variable accordigly in the loop. If then next step the recall is calculated using threshold which is still set to a value outside the predicted score range, the recall will be 0 (or 1, if high scores predict the negative class).

  ## calculate and plot the ROC
  roc <- data.frame (
           threshold = seq (min (score) - 1, max (score + 1), length.out = 100),
           tpr = NA, fpr = NA
         )
  
  for (i in 1 : 100) {
     threshold <- roc$threshold [i]
         roc$tpr [i] <- mean (score [class == "positive"] > threshold)
     roc$fpr [i] <- mean (score [class == "negative"] > threshold )
  }
  
  plot (roc$fpr, roc$tpr, col = rainbow (100))
  
  ## now the recall
  
  # **FORGET TO SET A SENSIBLE THRESHOLD VALUE**
  
  recall <-  mean (score [class == "positive"] > threshold)
  recall
  ## [1] 0
  

---

I used this R code to produce the graph:

require (ROCR)
perf <- performance (prediction (1:4, 1:4 > 2.5), "tpr", "fpr")
plot (perf, asp = 1)
points ([email protected] [[1]], [email protected][[1]], pch = 19, col = c(2, 1,1,1,1))