1
$\begingroup$

I have an XGBoost classifier and a dataset with 1,000 observations that I split 80% for training and 20% for testing. I’m trying to get confidence intervals for the ROC AUC metric. Before I learned about bootstrap confidence intervals, I would (method 1) train the model on the training set and report one AUC after running the model on the test set.

I tried following this code which was really helpful: bootsrap code But I’ don’t know if this is correct. In this code (method 2), it looks like 100% of the data is being used, a random sample of 800 observations are used for training, 200 for testing, and then this repeats, say 100 times, but each time with a different random sample of 800 observations for training, and the remaining 200 for testing. Is this a valid way to measure performance?

I’m confused because I saw this paper (method 3) where they mention “Model accuracy is reported on the test set, and 1000 bootstrapped samples were used to calculate 95% confidence intervals.” The way it’s written it sounds like they ignored the original training set and resampled the test data only 1,000 times and used that (in my case) 200 observations to train and test 1,000 times.

Can someone please explain step-by-step what is the right way to get bootstrapped confidence intervals? I want to generate confidence intervals correctly so that the AUC I’d traditionally get in the non-bootstrap method 1 falls within the range of the bootstrap CI from either method 2 or 3, but I’m not sure which method is the best representation of model performance.

$\endgroup$
3
$\begingroup$

With only 1000 observations, holding out a separate test set might not be the best approach. See this blog post by Frank Harrell for details. As he says:

... data splitting is an unstable method for validating models or classifiers, especially when the number of subjects is less than about 20,000 (fewer if signal:noise ratio is high). This is because were you to split the data again, develop a new model on the training sample, and test it on the holdout sample, the results are likely to vary significantly. Data splitting requires a significantly larger sample size than resampling to work acceptably well. ... Data splitting only has an advantage when the test sample is held by another researcher to ensure that the validation is unbiased.

So one could argue that in your case the "right way" to proceed would be to dispense completely with holding aside a test set, unless you have a high signal:noise ratio.

A good way to use the bootstrap for resampling is then to repeat your entire modeling process on, say, 1000 bootstrap samples of the data. Then apply each model to the entire original data set to evaluate its performance with your measure of interest.* Use the distribution of that measure among those 1000 models to estimate the confidence intervals (CI).

Frequentist CI are based on an assumption that you are sampling from a population in which the null hypothesis holds, while if you have found "significant" results you are presumably sampling from a population in which the null hypothesis doesn't hold. That can lead to problems, with several ways to deal with them. The boot.ci() function in R can return 4 different CI estimates. Think carefully about which is best in your case.

Also, although AUC is an easy to understand measure it isn't necessarily the best way to discriminate among models. See this page for links to other measures of model performance.


*The idea is that bootstrap resampling from the original data set represents the process of taking the original data set from the entire population. So evaluating the models based on the bootstrap samples against the original data set estimates how well the model based on the original data set would work on the entire population.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ (+1) Is this (more or less) method 2 in the OP? $\endgroup$ – Michael M Jun 27 at 16:20
  • 1
    $\begingroup$ @MichaelM method 2 in the OP seems to use the data excluded from the bootstrap sample to evaluate each of the 1000 models. The method I recommend is to use the entire original data set to evaluate each of the models. $\endgroup$ – EdM Jun 27 at 16:26
  • $\begingroup$ Ahhh, thanks for clarifying! $\endgroup$ – Michael M Jun 27 at 16:27
  • $\begingroup$ @EdM if I had, say, 100k observations and used a training and test set, when you say to “repeat the entire modeling process,” it would follow my “method 2,” correct? In other words, using resampled, say, 80k observations to train the model and 20k to test, gather the score and repeat with resampled 80k observations 1,000 times, and evaluate against the remaining 20k? I’m confused because if I applied your method, using a resampled 80k to train and then evaluate against the entire original dataset, most of the observations used for training will be scored, leading to biased results? $\endgroup$ – Insu Q Jun 27 at 16:46
  • $\begingroup$ @InsuQ Your method 2 would seem to be OK in that case. Then you could also evaluate bias due to validating on samples used for training pretty simply: for each model developed from a bootstrapped sample of the 80k training set, validate separately on the entire 80k training set and on the 20k test set. See how those 2 sets of validations compare. I suspect that any bias between them will be less than you fear. Note that some statistics like Shannon entropy are inherently biased even in random sampling; bootstrapping can help estimate that bias. $\endgroup$ – EdM Jun 27 at 21:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.