7
$\begingroup$

How does one calculate confidence intervals of cross-validated estimates?

For an epidemiological paper we use cat. and cont. NRI, IDI, and difference in C index for comparison of two Cox models. The reviewer suggested showing only cross-validated estimates and their 95% confidence intervals.

My ideas include taking the appropriate quantiles of the CV resamples, calculating the SE of those resamples and constructing Wald intervals, or bootstrapping the CI of the resamples' mean or median. But somehow these all seem phony.

| cite | improve this question | |
$\endgroup$
  • 1
    $\begingroup$ We went with cross-validating the confidence limits. It was computationally expensive. $\endgroup$ – miura Mar 12 '14 at 6:31
  • $\begingroup$ What exactly do you mean by "cross-validating the confidence limits"? $\endgroup$ – amoeba Mar 21 '14 at 10:39
  • 2
    $\begingroup$ Well, for each fold in each repetition of the cross-validation resampling, we not only calculated the maximum likelihood estimate of our statistic of interest, but also its confidence limits. We then took the average over all these replications for each the lower and upper confidence bound as well as the main estimate. In effect we cross-validated the confidence limits like any other statistic. $\endgroup$ – miura Mar 21 '14 at 12:38
5
$\begingroup$

For our credit risk paper on predicting loan defaults, a reviewer also suggested we produce confidence intervals for cross validation estimates and in particular recommended bootstrapping of the resampled mean.

Bootstrapped CIs were produced for risk ranking measures including the AUC, H-measure and the Kolmogorov-Smirnov (K-S) statistic. They were used to compare discrimination performance of two survival models - Mixture Cure, Cox with logistic regression.

It would be interesting to learn of other approaches to such CIs.

Tong, E.N.C., Mues, C. & Thomas, L.C. (2012) Mixture cure models in credit scoring: If and when borrowers default. European Journal of Operational Research, 218, (1), 132-139.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Hi @edward-tong, thanks for the link! In the first paragraph of the "Model building and validation" part, you said "To provide an approximate measure of variance for these measures, bootstrapping was applied to the resulting validation sample" from your CV 100-fold. Could you clarify how did you use bootstrap over the 100-fold resultset, with 1000 samples? $\endgroup$ – Bruno Ambrozio Apr 10 at 17:35
  • $\begingroup$ Hi @BrunoAmbrozio, we obtain model predictions using 100-fold CV. We then applied bootstrapping on this CV sample, obtaining a total of 1000 bootstrapped samples. Metrics of interest and corresponding CIs were then computed from these bootstrapped samples. $\endgroup$ – Edward Tong Jun 20 at 17:35
2
$\begingroup$

Recently I published a paper reporting mean and 95% confidence intervals for a number of performance statistics (accuracy, sensitivity, specificity etc) for a logistic regression model. We used 10 repetitions of 10 fold cross validation, taking the test set result for each fold produced 100 values for each performance statistic. If you can reasonably assume these values are independent then 95% confidence intervals can be calculated from these values. If you can't assume independence then bootstrapping as discussed above may be more appropriate.

| cite | improve this answer | |
$\endgroup$
  • 5
    $\begingroup$ BGreene, calculating confidence intervals from repeated k-fold cross validation not be a good idea according to some researchers: lirias.kuleuven.be/bitstream/123456789/346385/3/… A bias-corrected bootstrap confidence interval might be a better idea. $\endgroup$ – user41723 Mar 11 '14 at 18:27
  • 2
    $\begingroup$ BGreene, can you kindly give a link to your paper? $\endgroup$ – Serendipity Jan 31 '16 at 14:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.