Using a manual implementation of 5-fold cv for a binary classification problem, I calculate the AUC for each validation fold (using the predicted probabilities in each folds), and get scores of 0.870, 0.854, 0.886, 0.940, 0.921. I then calculate the AUC using all of the predicted probabilities, but get an AUC of 0.756. I get similar results via sklearn's implementation using cross_val_score and cross_val_predict.

Why might this be? How can a set of predicted probabilities (e.g. in the first validation fold) give me a high validation fold AUC but when combined with the other predicted probabilities give me a low overall AUC?

This seems related to a question asked here about why it is inappropriate to use the predictions from cross_validate_predict to compute metrics, but the explanation of the accepted answer (below) did not help my confusion per how come the AUC scores would differ:

The cross_val_score seems to say that it averages across all of the folds, while the cross_val_predict groups individual folds and distinct models but not all and therefore it won't necessarily generalize as well

See below for the code I used to get these results.

Manual implementation:

splitter = StratifiedKFold(n_splits = 5, shuffle = False)
predictions = pd.DataFrame({'actuals': y, 'probabilities': 0})

for i, (train_index, validate_index) in enumerate(splitter.split(X, y)):
    X_train, y_train, X_validate, y_validate = X.iloc[train_index], y.iloc[train_index], X.iloc[validate_index], y.iloc[validate_index]

    classifier.fit(X_train, y_train)

    probs = model.predict_proba(X_validate)[:,1]
    print(roc_auc_score(y_validate, probs))

    out = pd.DataFrame({'probabilities': probs}, index = X.index[validate_index])

auc = roc_auc_score(y_true = predictions.actuals.values, y_score = predictions.probabilities.values)

Sci-kit implementation:

scores = cross_val_score(classifier, X, y, scoring = 'roc_auc', cv = splitter)

predictions = cross_val_predict(classifier, X, y, method = 'predict_proba', cv = splitter)
roc_auc_score(y, predictions[:,1])

And so people could follow along I also tried to reproduce this discrepancy using the breast cancer dataset with logistic regression, but note I couldn't generate the discrepancy with such a simple dataset.

import numpy as np
import pandas as pd

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score, cross_val_predict
from sklearn.metrics import roc_auc_score

data = load_breast_cancer()
X = pd.DataFrame(data.data, columns = data.feature_names)
y = pd.Series(data.target)

classifier = LogisticRegression()

splitter = StratifiedKFold(n_splits = 5, shuffle = False)

scores = cross_val_score(classifier, X, y, scoring = 'roc_auc', cv = splitter)

predictions = cross_val_predict(classifier, X, y, method = 'predict_proba', cv = splitter)
roc_auc_score(y, predictions[:,1])
  • 1
    $\begingroup$ Have you compare the model coefficients within each fold? I'd assume that the model is changing a lot, overfitting for each fold. The overall model is calculated on all samples, so may be less likely to overfit. I note that there is a setting 'shuffle = False', which is alarming if I understand it correctly. If your data is organised in a biased way then not shuffling will lead to bias in the folds. What happens when you do shuffle? If that makes a difference you can test Monte Carlo k-fold where you redo the k-fold with reshuffling in each iteration. $\endgroup$ – ReneBt Jul 25 '19 at 4:52
  • $\begingroup$ That's a good thought, but the much lower overall AUC score in my manual implementation is actually just based on the same probabilities produced for each validation fold - I didn't create a new model for the overall validation set, but instead scored the probabilities for all the folds together with a single AUC score. To be explicit, I calculated the AUC score on {p_1}, and on {p_2}, ..., and on {p_5}, and then I calculated the AUC score on {p_1, p_2, ..., p_5}, where p_k designates the probabilities calculated for the kth validation fold $\endgroup$ – b.d Jul 25 '19 at 14:18
  • $\begingroup$ And that's great advice on the validation scheme - thank you. I intentionally set Shuffle = False because it made the discrepancy between the validation fold scores and the overall score pretty blatant here. Setting Shuffle = True does improve performance a lot! $\endgroup$ – b.d Jul 25 '19 at 14:20

Cross validation is a method for testing generalisation of model, not for calculating models themselves. It does this by holding out successive sets of samples, and in each fold it trains a model on the in-samples, tests on the out samples. The overall model is typically calculated from the entire training set rather than attempting to average across models.

If you do not have randomisation in your dataset (which is rare if it has not been not explicitly done), then (since if shuffled the folds will pick in a pattern), meaning there will be a bias between folds depending on the organisation of the dataset and how the folds are selected. A real-life visual example of this selection bias is the Moiré effect caused by using a pixelated camera detector to record a pixelated screen - if the two patterns do not exactly overlap then you get quite complex interference patterns between them. I suspect it was also the reason that old CGI movies optimised for viewing on large cinema screens didn't down-sample well for DVD (pseudo random shading has a structure).

If there are any systematic effects in sub-sampling for folds then there will be biased differences in the data submitted to the model builder in each fold. The outcome is that each fold will be creating completely different models that are not comparable.This means their probabilities are related to different things completely, so it is not surprising that their performance drops when an attempt to combine them is made.

Note that even a well randomised selection still presents different data to the model builder each time, so there will still be differences each time. This is why model scores on the training set are based on the overall model, not some aggregate of the folds.

The value of each fold is in the performance metrics used to assess the generalsiability, but this is based on the assumption of unbiased folding and comparability of models between folds. As you data shows it is important to ensure that the folding is done in order to ensure the best continuity between folds.This means as the dataset becomes more complex (e.g. lost of groups or subgroups) then the randomisation needs to be more complex to ensure fair distribution of important groups between folds. Other issues include ensuring that any repeated measurement from a sample is always kept within the same fold (otherwise the CV performance metrics are contaminated by reproducicibility variations, which are better handled separately).

  • $\begingroup$ Thanks for such a detailed response. I am still confused as to why mechanically* the union of 5 sets of probabilities gives such a different AUC than the AUC in each individual set, leaving aside whatever different underlying models were used to generate them. But you make it clear that mixing these sets of probabilities isn't such a meaningful thing to do anyways. Thanks! $\endgroup$ – b.d Sep 13 '19 at 19:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.