# How can the AUC on individual validation folds be much greater than the AUC on all validation data?

Using a manual implementation of 5-fold cv for a binary classification problem, I calculate the AUC for each validation fold (using the predicted probabilities in each folds), and get scores of 0.870, 0.854, 0.886, 0.940, 0.921. I then calculate the AUC using all of the predicted probabilities, but get an AUC of 0.756. I get similar results via sklearn's implementation using cross_val_score and cross_val_predict.

Why might this be? How can a set of predicted probabilities (e.g. in the first validation fold) give me a high validation fold AUC but when combined with the other predicted probabilities give me a low overall AUC?

This seems related to a question asked here about why it is inappropriate to use the predictions from cross_validate_predict to compute metrics, but the explanation of the accepted answer (below) did not help my confusion per how come the AUC scores would differ:

The cross_val_score seems to say that it averages across all of the folds, while the cross_val_predict groups individual folds and distinct models but not all and therefore it won't necessarily generalize as well

See below for the code I used to get these results.

Manual implementation:

splitter = StratifiedKFold(n_splits = 5, shuffle = False)
predictions = pd.DataFrame({'actuals': y, 'probabilities': 0})

for i, (train_index, validate_index) in enumerate(splitter.split(X, y)):
X_train, y_train, X_validate, y_validate = X.iloc[train_index], y.iloc[train_index], X.iloc[validate_index], y.iloc[validate_index]

classifier.fit(X_train, y_train)

probs = model.predict_proba(X_validate)[:,1]
print(roc_auc_score(y_validate, probs))

out = pd.DataFrame({'probabilities': probs}, index = X.index[validate_index])
predictions.update(out)

auc = roc_auc_score(y_true = predictions.actuals.values, y_score = predictions.probabilities.values)
print(auc)


Sci-kit implementation:

scores = cross_val_score(classifier, X, y, scoring = 'roc_auc', cv = splitter)
np.mean(scores)

predictions = cross_val_predict(classifier, X, y, method = 'predict_proba', cv = splitter)
roc_auc_score(y, predictions[:,1])


And so people could follow along I also tried to reproduce this discrepancy using the breast cancer dataset with logistic regression, but note I couldn't generate the discrepancy with such a simple dataset.

import numpy as np
import pandas as pd

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score, cross_val_predict
from sklearn.metrics import roc_auc_score

X = pd.DataFrame(data.data, columns = data.feature_names)
y = pd.Series(data.target)

classifier = LogisticRegression()

splitter = StratifiedKFold(n_splits = 5, shuffle = False)

scores = cross_val_score(classifier, X, y, scoring = 'roc_auc', cv = splitter)
scores

predictions = cross_val_predict(classifier, X, y, method = 'predict_proba', cv = splitter)
roc_auc_score(y, predictions[:,1])

• Have you compare the model coefficients within each fold? I'd assume that the model is changing a lot, overfitting for each fold. The overall model is calculated on all samples, so may be less likely to overfit. I note that there is a setting 'shuffle = False', which is alarming if I understand it correctly. If your data is organised in a biased way then not shuffling will lead to bias in the folds. What happens when you do shuffle? If that makes a difference you can test Monte Carlo k-fold where you redo the k-fold with reshuffling in each iteration. – ReneBt Jul 25 '19 at 4:52
• That's a good thought, but the much lower overall AUC score in my manual implementation is actually just based on the same probabilities produced for each validation fold - I didn't create a new model for the overall validation set, but instead scored the probabilities for all the folds together with a single AUC score. To be explicit, I calculated the AUC score on {p_1}, and on {p_2}, ..., and on {p_5}, and then I calculated the AUC score on {p_1, p_2, ..., p_5}, where p_k designates the probabilities calculated for the kth validation fold – b.d Jul 25 '19 at 14:18
• And that's great advice on the validation scheme - thank you. I intentionally set Shuffle = False because it made the discrepancy between the validation fold scores and the overall score pretty blatant here. Setting Shuffle = True does improve performance a lot! – b.d Jul 25 '19 at 14:20