# ROC curves from cross-validation are identical/overlaid and AUC is the same for each fold

UPDATE Confidence Intervals

I have an imbalanced dataset with around 200k instances and 50 predictors. The imbalance has a 4:1 ratio for the negative class (i.e class 0). In other words the negative class makes around 80% of the samples and the positive just 20% of the samples.

It's a binary classification problem where I have a target vector with 0's and 1's.

I have been trying to fit several classifiers like logistic regression and random forest.

I evaluate them with cross -validation skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=999)and ROC roc_curve from Python sklearn v.018

My Problem

My ROC curve for each validation fold are the same and I have no idea why. The AUC is also always absurdly good (0.9). Although the Precision-Recall Curve shows worse AUC=0.74 (which I think it's more accurate).

I tried following this example for ROC with cross-validation: http://lijiancheng0614.github.io/scikit-learn/auto_examples/model_selection/plot_roc_crossval.html#example-model-selection-plot-roc-crossval-py

ROC curves Logistic Regression

Precision Recall curves

ROC curves Random Forests

The question: Why does the performance of the model seems to be similar on each fold? shouldn't they differ at least slightly?

Code Below

clasifier = linear_model.LogisticRegression(class_weight = "balanced")
clasifier.fit(X,y)
fig, ax1 = plt.subplots(figsize=(12, 8))
mean_tpr = 0.0
mean_fpr = linspace(0, 1, 100)

skf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=999)

for i, (train_index, test_index) in enumerate(skf.split(X,y)):
# calculate the probability of each class assuming it to be positive
probas_ = classifier.fit(X[train_index], y[train_index]).predict_proba(X[test_index])
# Compute ROC curve and area under the curve
fpr, tpr, thresholds = roc_curve(y[test_index], probas_[:, 1], pos_label=1)
mean_tpr += interp(mean_fpr, fpr, tpr)
mean_tpr[0] = 0.0
roc_auc = auc(fpr, tpr)

plt.plot(fpr, tpr, lw=1, label='ROC fold %d (area = %0.2f)' % (i+1, roc_auc))

plt.plot([0, 1], [0, 1], '--', color=(0.6, 0.6, 0.6), label='Random', lw=2)

mean_tpr /= n_folds
mean_tpr[-1] = 1.0
mean_auc = auc(mean_fpr, mean_tpr)

plt.plot(mean_fpr, mean_tpr, 'k--',
label='Mean ROC (area = %0.2f)' % mean_auc, lw=3)
plt.xlim([-0.05, 1.05])
plt.ylim([-0.05, 1.05])
plt.xlabel('False Positive Rate (1- specificity)', fontsize=18)
plt.ylabel('True Positive Rate (sensitivity)', fontsize=18)

• What classifier are you using – Alex R. Mar 18 '17 at 16:21
• How large is your data set? – Calimo Mar 19 '17 at 7:14
• @Calimo 200k rows and 50 predictors/columns. – Roxanne Mar 19 '17 at 9:52
• @AlexR. logistic regression like this : clasifier = linear_model.LogisticRegression(class_weight = "balanced") – Roxanne Mar 19 '17 at 9:53
• Sorry I missed that. Now you can calculate the standard error and confidence intervals of your AUC with 200/k = 40 000. I bet it's below 0.01. – Calimo Mar 19 '17 at 10:09