The following code, which uses the function sklearn.model_selection.cross_validate and the scikit learn compatible XGBClassifier:

from sklearn.model_selection import cross_validate
from xgboost import XGBClassifier
from sklearn.model_selection import StratifiedShuffleSplit

cv_clf = XGBClassifier(gpu_id=0, tree_method='gpu_hist', n_estimators=300, max_depth=9)
cv = sklearn.model_selection.cross_validate(estimator=cv_clf, X=X, y=labels, scoring='accuracy', verbose=100, n_jobs=4, cv=10) 

produces results that are wildly different from the code below, even though they're pretty much supposed to do the same thing:

sss = StratifiedShuffleSplit(n_splits=10, test_size=0.1, random_state=42)

for train_indices, test_indices in sss.split(np.zeros(len(labels)), labels):
    X_train = dataset.data[train_indices]
    y_train = labels[train_indices]
    X_test = dataset.data[test_indices]
    y_test = labels[test_indices]
    age_train = age.values[train_indices]
    age_test = age.values[test_indices]
    X_train = np.concatenate([X_train, age_train], axis=1)
    X_test = np.concatenate([X_test, age_test], axis=1)
    cv_clf = XGBClassifier(gpu_id=0, tree_method='gpu_hist', n_estimators=300, n_jobs=4, max_depth=9)
    cv_clf.fit(X_train, y_train)
    preds = cv_clf.predict(X_test)
    accuracy = accuracy_score(preds, y_test)

X in the first code listing is the same as np.concatenate([dataset.data, age], axis=1)

In the first case, the accuracy is very low (40-70), in the second case, it's 88-91.

  • $\begingroup$ Both code chunks do slightly different things, the “by hand” approach is not a standard cross-validation. $\endgroup$
    – Tim
    May 16, 2021 at 11:06
  • $\begingroup$ @Tim in what sense is it not standard? $\endgroup$ May 16, 2021 at 11:24
  • $\begingroup$ @Tim Hmm I changed StratifiedShuffleSplit to StratifiedKFold in the manual approach and the accuracy dropped to similar values as those obtained from cross validate. But I'm a bit confused. How can the drop be so dramatic? $\endgroup$ May 16, 2021 at 11:37
  • $\begingroup$ @Tim I'm especially confused because before running the above experiments I ran experiments with GridSearch with the cv parameter set to 10, and that gave me good accuracy $\endgroup$ May 16, 2021 at 11:40
  • 1
    $\begingroup$ Are your rows in a particular order? StratifiedShuffleSplit obviously randomizes the order, whereas StratifiedKFold does not by default. Also, what's the size of your dataset? $\endgroup$ May 16, 2021 at 20:05

1 Answer 1


From the docs for cross_validate, parameter cv (as of v0.24.2):

For int/None inputs, if the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used. In all other cases, Fold [sic] is used. These splitters are instantiated with shuffle=False so the splits will be the same across calls.

The first sentence clarifies that your manual approach, with StratifiedShuffleSplit, is not the same as the cross-validation strategy of the first approach (k-fold). However, you would still expect the scores to be closer than what you're seeing.

The real problem is from the second sentence. Your cross_validate doesn't shuffle the data, while your manual approach does. If your data comes in a particular order, this can dramatically affect scores. You can fix this by passing an explicit cross-validation generator, e.g. cross_validate(..., cv=sss). Whether you want to shuffle the data depends on what the order of your rows means.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.