sklearn cross validation wildly different results from manual cross validation

Question

The following code, which uses the function sklearn.model_selection.cross_validate and the scikit learn compatible XGBClassifier:

from sklearn.model_selection import cross_validate
from xgboost import XGBClassifier
from sklearn.model_selection import StratifiedShuffleSplit

cv_clf = XGBClassifier(gpu_id=0, tree_method='gpu_hist', n_estimators=300, max_depth=9)
cv = sklearn.model_selection.cross_validate(estimator=cv_clf, X=X, y=labels, scoring='accuracy', verbose=100, n_jobs=4, cv=10) 

produces results that are wildly different from the code below, even though they're pretty much supposed to do the same thing:

sss = StratifiedShuffleSplit(n_splits=10, test_size=0.1, random_state=42)

for train_indices, test_indices in sss.split(np.zeros(len(labels)), labels):
    X_train = dataset.data[train_indices]
    y_train = labels[train_indices]
    X_test = dataset.data[test_indices]
    y_test = labels[test_indices]
    age_train = age.values[train_indices]
    age_test = age.values[test_indices]
    X_train = np.concatenate([X_train, age_train], axis=1)
    X_test = np.concatenate([X_test, age_test], axis=1)
    cv_clf = XGBClassifier(gpu_id=0, tree_method='gpu_hist', n_estimators=300, n_jobs=4, max_depth=9)
    cv_clf.fit(X_train, y_train)
    preds = cv_clf.predict(X_test)
    accuracy = accuracy_score(preds, y_test)
    print(accuracy)

X in the first code listing is the same as np.concatenate([dataset.data, age], axis=1)

In the first case, the accuracy is very low (40-70), in the second case, it's 88-91.

Answer 1

From the docs for cross_validate, parameter cv (as of v0.24.2):

For int/None inputs, if the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used. In all other cases, Fold [sic] is used. These splitters are instantiated with shuffle=False so the splits will be the same across calls.

The first sentence clarifies that your manual approach, with StratifiedShuffleSplit, is not the same as the cross-validation strategy of the first approach (k-fold). However, you would still expect the scores to be closer than what you're seeing.

The real problem is from the second sentence. Your cross_validate doesn't shuffle the data, while your manual approach does. If your data comes in a particular order, this can dramatically affect scores. You can fix this by passing an explicit cross-validation generator, e.g. cross_validate(..., cv=sss). Whether you want to shuffle the data depends on what the order of your rows means.