1
$\begingroup$

I have just started working with xgboost in python am pleased how its working and have just begun optimizing my model. Without going into to much detail, my model uses XGBClassifier to predict a binary variable Y and extract numeric features X and rank them on F-score rank. I was curious to the effect of random seed when splitting testing and training set so I decided to run a simulation of 10,000 models with a randomly generated seed between 1 and 10,000. The dataset has 20 variables X and about 3600 rows.

These are my results: Results

To me it seems as though prediction accuracy converges on 52% however by picking extreme outliers accuracy can exceed 56% (6316) or drop below 48% (987). It seems strange to me that when I used these seeds on the pima indians sets (very different to my original data) models improved or deteriorated in a similar fashion. What is going on here?

Are these improvements trivial? How can you rigorously test if the improvements you make to your model are no better than random?

$\endgroup$
2
  • 2
    $\begingroup$ +1 to Hristo's answer. I am commenting to stress this even more: Picking a convenient seed is one of the most sneaky ways of presenting over-optimistic results. If the choice of the random seed affects the final estimates then this means that the procedure in place is straight-up unreliable for the question at hand. Setting a random seed should be done for the purpose of reproducibility, not for the purpose of getting "good results". $\endgroup$
    – usεr11852
    Dec 24, 2018 at 17:56
  • $\begingroup$ To make a case to your point you should plot the acuracy on model 1 and acuracy on model2 as data points and then study if there exists a relation between them. Just saying models improved or deteriorated in similar fashion is not enough. $\endgroup$
    – Manuel
    Nov 2, 2021 at 14:50

3 Answers 3

7
$\begingroup$

The accuracy seem to follow gaussian distribution around 52%. You are essentially doing validation overfitting.

$\endgroup$
1
  • 1
    $\begingroup$ Good point, but few more words of explanation would be even better. $\endgroup$
    – Tim
    Dec 24, 2018 at 14:20
2
$\begingroup$

The high variability in performance depending on seed is an indication that you have a noisy validation strategy. A single train-test split is a terrible idea for small to moderate sized datasets. The variability could be reduced e.g. with repeated cross-validation. Note: you should never evaluate repeatedly on a true holdout test set or try different ways of splitting, so I'll call it a validation set. I guess, for a true holdout set, one thing you'd probably do for a small dataset is to think about whether it makes sense to stratify the split by outcome.

I'd also be highly suspicious of whether this means anything meaningful about some "training data being better" for a XGBoost model, as I also sort of expressed for random forest for a similar question. You may just be picking the validation set that is the easiest to fit.

$\endgroup$
0
$\begingroup$

I have a similar question. But now I think the increase of performance at certain random seed could be real, not overfit. I do the following experiment to prove it. In one of my modeling projects, I split the data into train/validation/test by 70%/15%/15% . Then similar to questioner's method, I keep a set of parameters of xgboost model same , only change the random seed. For each random seed, train the model with early stop, i.e. stop when performance stops increasing on valid dataset. The performance metric here is MAP (mean average precision). In the end , there were about 400 random seeds corresponding about 400 models with different validation data performance. At last, I apply all the models to test dataset, to computer test data MAP. The following is relation between validation MAP and test MAP. The shade is 95% confidence interval. They are significantly correlated tested by Pearson correlation test (correlation=0.356, p=4.13e-14). It should not be this case if it just overfit. This means if I tune random seed to select the best model on validation data, the model will likely (on average) perform better on test data. enter image description here The result is surprising. My explanation is as following. The random seed most likely effects the resulted xgboost model through the sampling of train data in the fitting process (I use colsample_bytree and subsample). So a random seed will determine a 'path' of trees that focus on different part (e.g. subset of columns) of training data. I use the term 'path' as a metaphor here as one tree in boost algorithm will effect the subsequence trees . A better model with better 'path' of different focuses on training data could be generalized if it reflects the intrinsic property of the data.

$\endgroup$
1
  • $\begingroup$ Can you replicate this on a secondary holdout set? I.e., 60%,15%,15% + 10%, where you repeat the same analysis on both the test 15% and the holdout 10%? Further, can you replicate this on any random train/validation/test split? $\endgroup$
    – runr
    Nov 8, 2021 at 16:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.