I have just started working with xgboost in python am pleased how its working and have just begun optimizing my model. Without going into to much detail, my model uses XGBClassifier to predict a binary variable Y and extract numeric features X and rank them on F-score rank. I was curious to the effect of random seed when splitting testing and training set so I decided to run a simulation of 10,000 models with a randomly generated seed between 1 and 10,000. The dataset has 20 variables X and about 3600 rows.

These are my results: Results

To me it seems as though prediction accuracy converges on 52% however by picking extreme outliers accuracy can exceed 56% (6316) or drop below 48% (987). It seems strange to me that when I used these seeds on the pima indians sets (very different to my original data) models improved or deteriorated in a similar fashion. What is going on here?

Are these improvements trivial? How can you rigorously test if the improvements you make to your model are no better than random?

  • $\begingroup$ +1 to Hristo's answer. I am commenting to stress this even more: Picking a convenient seed is one of the most sneaky ways of presenting over-optimistic results. If the choice of the random seed affects the final estimates then this means that the procedure in place is straight-up unreliable for the question at hand. Setting a random seed should be done for the purpose of reproducibility, not for the purpose of getting "good results". $\endgroup$ – usεr11852 Dec 24 '18 at 17:56

The accuracy seem to follow gaussian distribution around 52%. You are essentially doing validation overfitting.

  • $\begingroup$ Good point, but few more words of explanation would be even better. $\endgroup$ – Tim Dec 24 '18 at 14:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.