I have trained an XGBoost classifier for a binary classification. The situation I don't understand is when applying a k-folds cross validation on the model I get scores sensibly different than when doing a train/test split, even when this last process is repeated multiple times.
To know - on code:
- All done with Python
- I am using sklearn version 0.19.0
- I am using xgboost version 0.71
To start with, I've done a train/test split of the dataset, using a test size of 20%. My original dataset contains about 34000 items, the 2 classes are balanced so in this train/test I end up with about 3400 items in each class for the test set. I am using sklearn's
The accuracy I get when doing this is 82% (average accuracy across the two classes). If I repeat this procedure, always using a test size of 20% and making sure the random seed in the samples selection is different every time, I get all values compatible to this 82%. Note that classes in the splits are balanced.
The other procedure I've run has been using a k-fold cross validation, with k=5. This should effectively be equivalent to the procedure above as 5 folds will mean using a test set built from 20% of the data. Again I made sure that the folds are stratified for class balance. For this, I am using sklearn's cross_val_score, which as per docs returns the average accuracy in the case of a classifier.
The interesting thing is that in this case I get accuracy values all sensibly lower than 82%. I am wondering what is wrong in all this, and what is the difference between the two procedures that is driving these behaviours.
A note on data
I have applied the same procedures on one of the default sklearn datasets, and I don't see the difference. My dataset contains some NULL values, which XGBoost is resilient to.