10
$\begingroup$

How does one calculate confidence intervals of cross-validated estimates?

For an epidemiological paper we use cat. and cont. NRI, IDI, and difference in C index for comparison of two Cox models. The reviewer suggested showing only cross-validated estimates and their 95% confidence intervals.

My ideas include taking the appropriate quantiles of the CV resamples, calculating the SE of those resamples and constructing Wald intervals, or bootstrapping the CI of the resamples' mean or median. But somehow these all seem phony.

Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ We went with cross-validating the confidence limits. It was computationally expensive. $\endgroup$
    – miura
    Mar 12, 2014 at 6:31
  • $\begingroup$ What exactly do you mean by "cross-validating the confidence limits"? $\endgroup$
    – amoeba
    Mar 21, 2014 at 10:39
  • 3
    $\begingroup$ Well, for each fold in each repetition of the cross-validation resampling, we not only calculated the maximum likelihood estimate of our statistic of interest, but also its confidence limits. We then took the average over all these replications for each the lower and upper confidence bound as well as the main estimate. In effect we cross-validated the confidence limits like any other statistic. $\endgroup$
    – miura
    Mar 21, 2014 at 12:38

3 Answers 3

Reset to default
7
$\begingroup$

For our credit risk paper on predicting loan defaults, a reviewer also suggested we produce confidence intervals for cross validation estimates and in particular recommended bootstrapping of the resampled mean.

Bootstrapped CIs were produced for risk ranking measures including the AUC, H-measure and the Kolmogorov-Smirnov (K-S) statistic. They were used to compare discrimination performance of two survival models - Mixture Cure, Cox with logistic regression.

It would be interesting to learn of other approaches to such CIs.

Tong, E.N.C., Mues, C. & Thomas, L.C. (2012) Mixture cure models in credit scoring: If and when borrowers default. European Journal of Operational Research, 218, (1), 132-139.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Hi @edward-tong, thanks for the link! In the first paragraph of the "Model building and validation" part, you said "To provide an approximate measure of variance for these measures, bootstrapping was applied to the resulting validation sample" from your CV 100-fold. Could you clarify how did you use bootstrap over the 100-fold resultset, with 1000 samples? $\endgroup$
    – Bruno Ambrozio
    Apr 10, 2020 at 17:35
  • $\begingroup$ Hi @BrunoAmbrozio, we obtain model predictions using 100-fold CV. We then applied bootstrapping on this CV sample, obtaining a total of 1000 bootstrapped samples. Metrics of interest and corresponding CIs were then computed from these bootstrapped samples. $\endgroup$
    – Edward Tong
    Jun 20, 2020 at 17:35
4
$\begingroup$

If you can't assume independence of the data splits (which in many scenarios you can't), here's a method that allows for the computation of "valid" confidence intervals around your error. It was recently published by Stanford (2021) so there still aren't python packages, but they did create an R package.

I was interested in the topic so I made a less technical writeup, but the paper tells the full story.

Paper Info (in case the link dies):

  • Name: Cross-validation: what does it estimate and how well does it do it?
  • Authors: Stephen Bates, Trevor Hastie, and Robert Tibshirani
  • Year: 2021
  • Key conclusions: "We have made two main contributions. First, we discussed point estimates of prediction error via subsampling techniques. Our primary result is that common estimates of prediction error—cross-validation, bootstrap, data splitting, and covariance penalties—cannot be viewed as estimates of the prediction error of the final model fit on the whole data. ... Secondly, we discuss inference for cross-validation, deriving an estimator for the MSE of the CV point estimate, nested CV."
Improve this answer
$\endgroup$
0
2
$\begingroup$

Recently I published a paper reporting mean and 95% confidence intervals for a number of performance statistics (accuracy, sensitivity, specificity etc) for a logistic regression model. We used 10 repetitions of 10 fold cross validation, taking the test set result for each fold produced 100 values for each performance statistic. If you can reasonably assume these values are independent then 95% confidence intervals can be calculated from these values. If you can't assume independence then bootstrapping as discussed above may be more appropriate.

Improve this answer
$\endgroup$
3
  • 5
    $\begingroup$ BGreene, calculating confidence intervals from repeated k-fold cross validation not be a good idea according to some researchers: lirias.kuleuven.be/bitstream/123456789/346385/3/… A bias-corrected bootstrap confidence interval might be a better idea. $\endgroup$
    – user41723
    Mar 11, 2014 at 18:27
  • 2
    $\begingroup$ BGreene, can you kindly give a link to your paper? $\endgroup$
    – Serendipity
    Jan 31, 2016 at 14:20
  • 1
    $\begingroup$ @user41723 your reference to the work of G. Vanwinckelen and H. Blockeel is indeed important. However, that is not what BGreene was doing. In the referenced paper you cite the authors criticise variance reduction technique by repeating k-fold CV. In essence, they say that taking the average error (or accuracy) from each k-fold CV and then calculating a new average over the number of repetitions (average of averages) does not necessarily say something about the generalization error. They then go further by claiming that some researchers calculate the variance (or CI) only over the repetitions $\endgroup$
    – JJR
    Nov 29, 2020 at 11:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.