I used sklearn to compute roc_auc_score
for a dataset of 72 instances.
The accuracy was at 97% (2 misclassifications), but the ROC AUC score was 1.0. How is this possible? I would think that even one misclassification should have dropped the score to slightly below 1.0.
# Python 3.6.4
# numpy==1.14.3
# scikit-learn==0.19.1
# scipy==1.1.0
from sklearn import metrics
import numpy as np
y_true = np.array([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0])
y_prob = np.array([0.0, 0.1, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.1, 0.0, 0.0, 0.0, 0.0, 0.7, 0.5, 0.0, 0.0, 0.0, 0.0, 0.0, 0.1, 0.1, 0.0, 0.0, 0.9, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.0, 0.1, 0.1, 0.0, 0.0, 0.7, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.1, 0.1, 0.0, 0.9, 0.0, 0.0, 0.4, 0.0, 0.0, 0.0])
# Show which actuals do not match their expected probabilities
for index, (actual, predicted_prob) in enumerate(zip(y_true, y_prob)):
if (actual == 1 and predicted_prob <= 0.5) or (actual == 0 and predicted_prob > 0.5):
print (f'Mismatch at index {index}. Actual={actual}, predicted_prob={predicted_prob}')
rocauc = metrics.roc_auc_score(y_true, y_prob)
print (f'ROCAUC: {rocauc}')
# Outputs:
# Mismatch at index 14. Actual=1.0, predicted_prob=0.5
# Mismatch at index 68. Actual=1.0, predicted_prob=0.4
# ROCAUC: 1.0
Then I debugged the score computation itself and looked at the coordinate ROC output.
# In sklearn/metrics/ranking.py, line 271:
--> 271 fpr, tpr, tresholds = roc_curve(y_true, y_score,
272 sample_weight=sample_weight)
ipdb> fpr
array([0. , 0. , 0. , 0.18181818, 1. ])
ipdb> tpr
array([0.33333333, 0.66666667, 1. , 1. , 1. ])
ipdb> tresholds
array([0.9, 0.7, 0.4, 0.1, 0. ])
# coords = []
# for x, y in zip(fpr, tpr):
# coords.append((x, y))
ipdb> pp coords
[(0.0, 0.3333333333333333),
(0.0, 0.6666666666666666),
(0.0, 1.0),
(0.18181818181818182, 1.0),
(1.0, 1.0)]
All of those coordinates are on x=0 or y=1, meaning ROCAUC is showing 1.0. The only plausible explanation I can think of is that if the alg had more fpr/tpr points, it would show a very tight curve that never reaches x,y = (0, 1), and the ROCAUC would be close to 1, but not exactly 1. Is that a reasonable interpretation or am I missing something?