Confidence intervals for cross-validated statistics How does one calculate confidence intervals of cross-validated estimates?
For an epidemiological paper we use cat. and cont. NRI, IDI, and difference in C index for comparison of two Cox models. The reviewer suggested showing only cross-validated estimates and their 95% confidence intervals.
My ideas include taking the appropriate quantiles of the CV resamples, calculating the SE of those resamples and constructing Wald intervals, or bootstrapping the CI of the resamples' mean or median. But somehow these all seem phony.
 A: For our credit risk paper on predicting loan defaults, a reviewer also suggested we produce confidence intervals for cross validation estimates and in particular recommended bootstrapping of the resampled mean.
Bootstrapped CIs were produced for risk ranking measures including the AUC, H-measure and the Kolmogorov-Smirnov (K-S) statistic. They were used to compare discrimination performance of two survival models - Mixture Cure, Cox with logistic regression.
It would be interesting to learn of other approaches to such CIs.
Tong, E.N.C., Mues, C. & Thomas, L.C. (2012) Mixture cure models in credit scoring: If and when borrowers default. European Journal of Operational Research, 218, (1), 132-139.
A: If you can't assume independence of the data splits (which in many scenarios you can't), here's a method that allows for the computation of "valid" confidence intervals around your error. It was recently published by Stanford (2021) so there still aren't python packages, but they did create an R package.
I was interested in the topic so I made a less technical writeup, but the paper tells the full story.
Paper Info (in case the link dies):

*

*Name: Cross-validation: what does it estimate and how well does it do it?

*Authors: Stephen Bates, Trevor Hastie, and Robert Tibshirani

*Year: 2021

*Key conclusions: "We have made two main contributions. First, we discussed point estimates of prediction error via subsampling
techniques. Our primary result is that common estimates of prediction error—cross-validation, bootstrap, data splitting, and covariance penalties—cannot be viewed as estimates of the prediction error of the final model fit on the whole data. ... Secondly, we discuss inference for cross-validation, deriving an estimator for the MSE of the CV point estimate, nested CV."

A: Recently I published a paper reporting mean and 95% confidence intervals for a number of performance statistics (accuracy, sensitivity, specificity etc) for a logistic regression model.
We used 10 repetitions of 10 fold cross validation, taking the test set result for each fold produced 100 values for each performance statistic.
If you can reasonably assume these values are independent then 95% confidence intervals can be calculated from these values. If you can't assume independence then bootstrapping as discussed above may be more appropriate.
