What are the steps for generating bootstrap confidence intervals?

Question

I have an XGBoost classifier and a dataset with 1,000 observations that I split 80% for training and 20% for testing. I’m trying to get confidence intervals for the ROC AUC metric. Before I learned about bootstrap confidence intervals, I would (method 1) train the model on the training set and report one AUC after running the model on the test set.

I tried following this code which was really helpful: bootsrap code But I’ don’t know if this is correct. In this code (method 2), it looks like 100% of the data is being used, a random sample of 800 observations are used for training, 200 for testing, and then this repeats, say 100 times, but each time with a different random sample of 800 observations for training, and the remaining 200 for testing. Is this a valid way to measure performance?

I’m confused because I saw this paper (method 3) where they mention “Model accuracy is reported on the test set, and 1000 bootstrapped samples were used to calculate 95% confidence intervals.” The way it’s written it sounds like they ignored the original training set and resampled the test data only 1,000 times and used that (in my case) 200 observations to train and test 1,000 times.

Can someone please explain step-by-step what is the right way to get bootstrapped confidence intervals? I want to generate confidence intervals correctly so that the AUC I’d traditionally get in the non-bootstrap method 1 falls within the range of the bootstrap CI from either method 2 or 3, but I’m not sure which method is the best representation of model performance.

Answer 1

With only 1000 observations, holding out a separate test set might not be the best approach. See this blog post by Frank Harrell for details. As he says:

... data splitting is an unstable method for validating models or classifiers, especially when the number of subjects is less than about 20,000 (fewer if signal:noise ratio is high). This is because were you to split the data again, develop a new model on the training sample, and test it on the holdout sample, the results are likely to vary significantly. Data splitting requires a significantly larger sample size than resampling to work acceptably well. ... Data splitting only has an advantage when the test sample is held by another researcher to ensure that the validation is unbiased.

So one could argue that in your case the "right way" to proceed would be to dispense completely with holding aside a test set, unless you have a high signal:noise ratio.

A good way to use the bootstrap for resampling is then to repeat your entire modeling process on, say, 1000 bootstrap samples of the data. Then apply each model to the entire original data set to evaluate its performance with your measure of interest.* Use the distribution of that measure among those 1000 models to estimate the confidence intervals (CI).

Frequentist CI are based on an assumption that you are sampling from a population in which the null hypothesis holds, while if you have found "significant" results you are presumably sampling from a population in which the null hypothesis doesn't hold. That can lead to problems, with several ways to deal with them. The boot.ci() function in R can return 4 different CI estimates. Think carefully about which is best in your case.

Also, although AUC is an easy to understand measure it isn't necessarily the best way to discriminate among models. See this page for links to other measures of model performance.

---

*The idea is that bootstrap resampling from the original data set represents the process of taking the original data set from the entire population. So evaluating the models based on the bootstrap samples against the original data set estimates how well the model based on the original data set would work on the entire population.