How is a ROCAUC=1.0 possible with imperfect accuracy? [duplicate]

Question

I used sklearn to compute roc_auc_score for a dataset of 72 instances. The accuracy was at 97% (2 misclassifications), but the ROC AUC score was 1.0. How is this possible? I would think that even one misclassification should have dropped the score to slightly below 1.0.

# Python 3.6.4
# numpy==1.14.3
# scikit-learn==0.19.1
# scipy==1.1.0

from sklearn import metrics
import numpy as np

y_true = np.array([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0])
y_prob = np.array([0.0, 0.1, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.1, 0.0, 0.0, 0.0, 0.0, 0.7, 0.5, 0.0, 0.0, 0.0, 0.0, 0.0, 0.1, 0.1, 0.0, 0.0, 0.9, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.0, 0.1, 0.1, 0.0, 0.0, 0.7, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.1, 0.1, 0.0, 0.9, 0.0, 0.0, 0.4, 0.0, 0.0, 0.0])

# Show which actuals do not match their expected probabilities
for index, (actual, predicted_prob) in enumerate(zip(y_true, y_prob)):
  if (actual == 1 and predicted_prob <= 0.5) or (actual == 0 and predicted_prob > 0.5):
    print (f'Mismatch at index {index}. Actual={actual}, predicted_prob={predicted_prob}')


rocauc = metrics.roc_auc_score(y_true, y_prob)
    print (f'ROCAUC: {rocauc}')

# Outputs:
# Mismatch at index 14. Actual=1.0, predicted_prob=0.5
# Mismatch at index 68. Actual=1.0, predicted_prob=0.4
# ROCAUC: 1.0

Then I debugged the score computation itself and looked at the coordinate ROC output.

# In sklearn/metrics/ranking.py, line 271:
--> 271         fpr, tpr, tresholds = roc_curve(y_true, y_score,
    272                                         sample_weight=sample_weight)

ipdb> fpr
array([0.        , 0.        , 0.        , 0.18181818, 1.        ])
ipdb> tpr
array([0.33333333, 0.66666667, 1.        , 1.        , 1.        ])
ipdb> tresholds
array([0.9, 0.7, 0.4, 0.1, 0. ])

# coords = []
# for x, y in zip(fpr, tpr):
#   coords.append((x, y))

ipdb> pp coords
[(0.0, 0.3333333333333333),
 (0.0, 0.6666666666666666),
 (0.0, 1.0),
 (0.18181818181818182, 1.0),
 (1.0, 1.0)]

All of those coordinates are on x=0 or y=1, meaning ROCAUC is showing 1.0. The only plausible explanation I can think of is that if the alg had more fpr/tpr points, it would show a very tight curve that never reaches x,y = (0, 1), and the ROCAUC would be close to 1, but not exactly 1. Is that a reasonable interpretation or am I missing something?

Answer 1

ROC AUC and the $c$-statistic are equivalent, and measure the probability that a randomly-chosen positive sample is ranked higher than a randomly-chosen negative sample. If all positives have score 0.49 and all negatives have score 0.48, then the ROC AUC is 1.0 because of this property. This can lead to counter-intuitive results. In this hypothetical, the accuracy, using the rule of a 0.5 cutoff, is 0.0 because all of the predictions are below 0.5!

In your data, there's a sample with prediction 0.4 but label 1.0; this is the sample with the lowest score and label 1.0. This sample, and the sample with label 1.0 and score 0.5, are decreasing your accuracy. But the highest score for the samples with label 0.0 is 0.1, so we know that we are dealing with the case of a perfect ROC AUC because 0.1 is less than 0.4.

I've found this book to be a good resource for information about ROC curves: Wojtek J. Krzanowski (Author) & David J. Hand. ROC Curves for Continuous Data