How to report accuracy of predicted classifications with CI when using k-fold CV?

Question

I modelled a binary response against several explanatory variables using logistic regression. There were various combinations of the explanatory variables that produced reasonable fits. I would like to validate and compare the models by using 10-fold CV by comparing the mean positive prediction rates.

When doing such a comparison, is it enough to report the mean prediction rate and provide the standard deviation of the accuracy across the ten folds? Or is it possible to report confidence intervals?

Answer 1

It is certainly possible to report the confidence intervals of a performance estimate (e.g. classification accuracy) by using the classification accuracies produced by the test set for each fold in cross-validation. Often it is better to report the mean classification accuracy and confidence intervals obtained through repeated cross-validation (rather than a single k-fold cross-validation) A related answer is here: Is it valid to apply a t-test on scores obtained by cross validation?

Answer 2

There are several things to be considered here.

You have (at least) 2 sources of random uncertainty in a cross validation setting:

• The number of test cases you have is finite. You can calcuate confidence intervals for proportions (in R, e.g. binom::binom.confint shows you a bunch of different methods and approximations).
• In the end of the day, this still stays finite, even if you run large numbers of iterations/repetitions of the cross validation.
• The 2nd source of random uncertainty is model instability. You train a whole lot of "surrogate models" that are assumed to be equvivalent. However, this may not be the case.
• Here, the iterations help: you can measure the variation introduced by instability if you look at the variation of predictions for the same case by different surrogate models (where the test case was of course always excluded from training).

Measuring variance over the iterations of the cross validation will not (or only partly - does anyone know, is there literature about this?) cover the uncertainty due to finite test sample size: if the models are perfectly stable, all surrogate models will yield the same prediction for any given test case. Then there will be no variance between the iterations. But the results are still subject to the finite-test-sample-size type of random uncertainty.

When calculating the confidence interval for such a proportion (using the number of independent samples), the result is usually quite shocking (at least in my field, where we often are happy if tens of cases are available...)

That is also the reason why model comparison based on proportions is not easy at all. You'll want to set this up in a paired way. Otherwise you'd need crazy numbers of test cases.