I have just started working with xgboost in python am pleased how its working and have just begun optimizing my model. Without going into to much detail, my model uses XGBClassifier to predict a binary variable Y and extract numeric features X and rank them on F-score rank. I was curious to the effect of random seed when splitting testing and training set so I decided to run a simulation of 10,000 models with a randomly generated seed between 1 and 10,000. The dataset has 20 variables X and about 3600 rows.
To me it seems as though prediction accuracy converges on 52% however by picking extreme outliers accuracy can exceed 56% (6316) or drop below 48% (987). It seems strange to me that when I used these seeds on the pima indians sets (very different to my original data) models improved or deteriorated in a similar fashion. What is going on here?
Are these improvements trivial? How can you rigorously test if the improvements you make to your model are no better than random?