XGBoost - "Optimizing Random Seed"

Question

I have just started working with xgboost in python am pleased how its working and have just begun optimizing my model. Without going into to much detail, my model uses XGBClassifier to predict a binary variable Y and extract numeric features X and rank them on F-score rank. I was curious to the effect of random seed when splitting testing and training set so I decided to run a simulation of 10,000 models with a randomly generated seed between 1 and 10,000. The dataset has 20 variables X and about 3600 rows.

These are my results:

To me it seems as though prediction accuracy converges on 52% however by picking extreme outliers accuracy can exceed 56% (6316) or drop below 48% (987). It seems strange to me that when I used these seeds on the pima indians sets (very different to my original data) models improved or deteriorated in a similar fashion. What is going on here?

Are these improvements trivial? How can you rigorously test if the improvements you make to your model are no better than random?

Answer 1

The accuracy seem to follow gaussian distribution around 52%. You are essentially doing validation overfitting.

Answer 2

The high variability in performance depending on seed is an indication that you have a noisy validation strategy. A single train-test split is a terrible idea for small to moderate sized datasets. The variability could be reduced e.g. with repeated cross-validation. Note: you should never evaluate repeatedly on a true holdout test set or try different ways of splitting, so I'll call it a validation set. I guess, for a true holdout set, one thing you'd probably do for a small dataset is to think about whether it makes sense to stratify the split by outcome.

I'd also be highly suspicious of whether this means anything meaningful about some "training data being better" for a XGBoost model, as I also sort of expressed for random forest for a similar question. You may just be picking the validation set that is the easiest to fit.

Answer 3

I have a similar question. But now I think the increase of performance at certain random seed could be real, not overfit. I do the following experiment to prove it. In one of my modeling projects, I split the data into train/validation/test by 70%/15%/15% . Then similar to questioner's method, I keep a set of parameters of xgboost model same , only change the random seed. For each random seed, train the model with early stop, i.e. stop when performance stops increasing on valid dataset. The performance metric here is MAP (mean average precision). In the end , there were about 400 random seeds corresponding about 400 models with different validation data performance. At last, I apply all the models to test dataset, to computer test data MAP. The following is relation between validation MAP and test MAP. The shade is 95% confidence interval. They are significantly correlated tested by Pearson correlation test (correlation=0.356, p=4.13e-14). It should not be this case if it just overfit. This means if I tune random seed to select the best model on validation data, the model will likely (on average) perform better on test data. The result is surprising. My explanation is as following. The random seed most likely effects the resulted xgboost model through the sampling of train data in the fitting process (I use colsample_bytree and subsample). So a random seed will determine a 'path' of trees that focus on different part (e.g. subset of columns) of training data. I use the term 'path' as a metaphor here as one tree in boost algorithm will effect the subsequence trees . A better model with better 'path' of different focuses on training data could be generalized if it reflects the intrinsic property of the data.