I have trained an XGBoost classifier for a binary classification. The situation I don't understand is when applying a k-folds cross validation on the model I get scores sensibly different than when doing a train/test split, even when this last process is repeated multiple times.

To know - on code:

  • All done with Python
  • I am using sklearn version 0.19.0
  • I am using xgboost version 0.71


To start with, I've done a train/test split of the dataset, using a test size of 20%. My original dataset contains about 34000 items, the 2 classes are balanced so in this train/test I end up with about 3400 items in each class for the test set. I am using sklearn's train_test_split.

The accuracy I get when doing this is 82% (average accuracy across the two classes). If I repeat this procedure, always using a test size of 20% and making sure the random seed in the samples selection is different every time, I get all values compatible to this 82%. Note that classes in the splits are balanced.

The other procedure I've run has been using a k-fold cross validation, with k=5. This should effectively be equivalent to the procedure above as 5 folds will mean using a test set built from 20% of the data. Again I made sure that the folds are stratified for class balance. For this, I am using sklearn's cross_val_score, which as per docs returns the average accuracy in the case of a classifier.

The interesting thing is that in this case I get accuracy values all sensibly lower than 82%. I am wondering what is wrong in all this, and what is the difference between the two procedures that is driving these behaviours.

A note on data

I have applied the same procedures on one of the default sklearn datasets, and I don't see the difference. My dataset contains some NULL values, which XGBoost is resilient to.

  • $\begingroup$ I actually seem to have solved this. The difference is made by the fact that a shuffle=True parameter is applied by default to train_test_split (as should be) but a shuffle=False is default on cross validation. Apparently my dataset exhibits dependency on order. $\endgroup$ – mar tin May 9 '18 at 12:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.