How did these researchers determine the confidence interval of the AUROC using resampling but without retraining the model?

Question

In this Nature article backed by Google, the investigators develop then externally validate a deep learning model for predicting lung cancer using CT scans. In their internal validation results, we can see they included 95% confidence intervals for their AUROC:

In their methods, they state:

All confidence intervals were computed based on the percentiles of 1,000 random resamplings (bootstraps) of the data. Confidence intervals for differences were derived by computing the metric of interest and then computing a reader–model difference on each bootstrap. P values for sensitivity and specificity comparisons were computed using a standard permutation test using 10,000 random resamplings of the data

When I read up how to obtain confidence intervals using the bootstrap method, what I understand is that the model must be retrained for every single bootstrap, and that the statistic is calculated for each retrained model (and the model is applied to the original pre-bootstrapped data). E.g. reference. This implies that Google retrained their deep learning model on a bootstrap of the training sample 1000 times to obtain these intervals. Seems like a lot of computation, but its Google, so OK. So far so good.

What I do not understand is how they obtained the confidence interval for the AUROC in their external validation study that used observations from another center:

They explicitly state in their article

The model was not trained or tuned using this dataset.

In the description of the figure, they say:

AUC curve for the independent data test set with n = 1,739 cases using a two-sided permutation test with 10,000 random resamplings of the data.

I've looked up how to do a permutation test, but I can't find a reference that shows the exact procedure to follow when using it to calculate a confidence interval for the external validity of a prediction model.

What is the generic procedure (implementation-wise) to obtain this confidence interval?

I can imagine two potential procedures:

Procedure 1:

1. Resample the test sample (with replacement, i.e. obtain a bootstrap)
2. Apply prediction model to obtain risk estimates
3. Calculate statistic
4. Repeat 1-3 n times then use the 0.025 and 0.975 percentiles of the statistic to obtain the 95% CI.

Procedure 2:

1. Calculate the AUC on the original test sample (call it AUROC_original)
2. Randomly permute the labels of the test sample to break the relationship between the features and the labels
3. Apply the model on this sample and obtain the AUROC (AUROC_i)
4. Repeat 2-3 n times and then use the distribution of all n AUROC_i's (which I assume is the distribution of the null hypothesis, that is, the possible AUROC values we would get if the model was useless) to infer the distribution around AUROC_original. No idea if this is valid or how it would be done. I assume you can't just add the difference between the 50th percentile and AUROC_original since the AUROC is bounded between zero and one. Maybe you would need to use the properties of e.g. the binomial distribution?

Any help (with references) would be appreciated! My goal is to be able to create confidence intervals for the statistics of a machine learning model that is validated on a large external sample. I do not want to retrain the model on this external sample because I'd like to understand how well my model would perform were it generalized there, given that the parameters of the model are not subject to change.

Another use case is validating the performance of a proprietary model that cannot be retrained with local data. Imagine a scenario where you need the confidence interval of various statistics like the Brier score, AUROC, AUPRC to be able to compare two proprietary models.

Answer 1

I think (but am not sure -- for that you may have to ask the authors) that it may be simpler than what you think. Even an identical model has a different AUROC value depending on the dataset used to calculate it.

Suppose we have some dataset $\{x_1, \ldots, x_n\}$ and some model $M$. Then, we can calculate its AUROC over this dataset:

$$ \text{AUROC}(x_1, \ldots, x_n | M).$$

However, we can also calculate the AUROC over another dataset $\{y_1, \ldots y_k\}$:

$$ \text{AUROC}(y_1, \ldots, y_k | M).$$

This gives a different AUROC value for the same model. I have glanced over the article without reading it in detail, and what I thought the authors had done was:

1. Use some training dataset $D_{tr} = \{x_1, \ldots, x_N\}$ to train a model $M$.
2. Calculate the AUROC across the training dataset: $$\text{AUROC}_{tr} = \text{AUROC}(x_1, \ldots, x_n | M).$$
3. Create 10,000 bootstrap samples $(D_{tr, i}^*)_{i=1}^{10000}$ from the training dataset -- i.e. for each $i$, $D_{tr,i}^* = \{x_{i, 1}^*, \ldots x_{i, n}^*\}$ is another length $N$ sample created by sampling with replacement from $D_{tr}$.
4. For each $D_{tr, i}^*$, calculate the AUROC given the original model $M$: $$\text{AUROC}_{tr, i}^* = \text{AUROC}(x_{i, 1}^*, \ldots x_{i, n}^* | M).$$ These are used to construct the confidence intervals for the training AUROC. I think, in the paper, they just used the 250th and 9,750th highest values to get a 95% confidence interval.
5. Given some validation dataset $\{y_1, \ldots, y_k\}$, repeat steps 2-4 to obtain the validation AUROCs: $$\text{AUROC}_{val} = \text{AUROC}(y_1, \ldots y_k | M).$$ $$\text{AUROC}_{val, i}^* = \text{AUROC}(y_{i, 1}^*, \ldots y_{i, k}^* | M).$$ Use the bottom ones to obtain confidence intervals.

This can all be done without retraining the model (which I find no mention of in my brief scan of the paper). It's pretty much the first procedure you suggest, except that the model is never retrained even on the bootstrap training samples.

Does that sound plausible to you? Of course, you can always ask the authors! :)