In this Nature article backed by Google, the investigators develop then externally validate a deep learning model for predicting lung cancer using CT scans. In their internal validation results, we can see they included 95% confidence intervals for their AUROC:
In their methods, they state:
All confidence intervals were computed based on the percentiles of 1,000 random resamplings (bootstraps) of the data. Confidence intervals for differences were derived by computing the metric of interest and then computing a reader–model difference on each bootstrap. P values for sensitivity and specificity comparisons were computed using a standard permutation test using 10,000 random resamplings of the data
When I read up how to obtain confidence intervals using the bootstrap method, what I understand is that the model must be retrained for every single bootstrap, and that the statistic is calculated for each retrained model (and the model is applied to the original pre-bootstrapped data). E.g. reference. This implies that Google retrained their deep learning model on a bootstrap of the training sample 1000 times to obtain these intervals. Seems like a lot of computation, but its Google, so OK. So far so good.
What I do not understand is how they obtained the confidence interval for the AUROC in their external validation study that used observations from another center:
They explicitly state in their article
The model was not trained or tuned using this dataset.
In the description of the figure, they say:
AUC curve for the independent data test set with n = 1,739 cases using a two-sided permutation test with 10,000 random resamplings of the data.
I've looked up how to do a permutation test, but I can't find a reference that shows the exact procedure to follow when using it to calculate a confidence interval for the external validity of a prediction model.
What is the generic procedure (implementation-wise) to obtain this confidence interval?
I can imagine two potential procedures:
- Resample the test sample (with replacement, i.e. obtain a bootstrap)
- Apply prediction model to obtain risk estimates
- Calculate statistic
- Repeat 1-3 n times then use the 0.025 and 0.975 percentiles of the statistic to obtain the 95% CI.
- Calculate the AUC on the original test sample (call it AUROC_original)
- Randomly permute the labels of the test sample to break the relationship between the features and the labels
- Apply the model on this sample and obtain the AUROC (AUROC_i)
- Repeat 2-3 n times and then use the distribution of all n AUROC_i's (which I assume is the distribution of the null hypothesis, that is, the possible AUROC values we would get if the model was useless) to infer the distribution around AUROC_original. No idea if this is valid or how it would be done. I assume you can't just add the difference between the 50th percentile and AUROC_original since the AUROC is bounded between zero and one. Maybe you would need to use the properties of e.g. the binomial distribution?
Any help (with references) would be appreciated! My goal is to be able to create confidence intervals for the statistics of a machine learning model that is validated on a large external sample. I do not want to retrain the model on this external sample because I'd like to understand how well my model would perform were it generalized there, given that the parameters of the model are not subject to change.
Another use case is validating the performance of a proprietary model that cannot be retrained with local data. Imagine a scenario where you need the confidence interval of various statistics like the Brier score, AUROC, AUPRC to be able to compare two proprietary models.