I have an XGBoost classifier and a dataset with 1,000 observations that I split 80% for training and 20% for testing. I’m trying to get confidence intervals for the ROC AUC metric. Before I learned about bootstrap confidence intervals, I would (method 1) train the model on the training set and report one AUC after running the model on the test set.
I tried following this code which was really helpful: bootsrap code But I’ don’t know if this is correct. In this code (method 2), it looks like 100% of the data is being used, a random sample of 800 observations are used for training, 200 for testing, and then this repeats, say 100 times, but each time with a different random sample of 800 observations for training, and the remaining 200 for testing. Is this a valid way to measure performance?
I’m confused because I saw this paper (method 3) where they mention “Model accuracy is reported on the test set, and 1000 bootstrapped samples were used to calculate 95% confidence intervals.” The way it’s written it sounds like they ignored the original training set and resampled the test data only 1,000 times and used that (in my case) 200 observations to train and test 1,000 times.
Can someone please explain step-by-step what is the right way to get bootstrapped confidence intervals? I want to generate confidence intervals correctly so that the AUC I’d traditionally get in the non-bootstrap method 1 falls within the range of the bootstrap CI from either method 2 or 3, but I’m not sure which method is the best representation of model performance.