There are several things to be considered here.
You have (at least) 2 sources of random uncertainty in a cross validation setting:
- The number of test cases you have is finite. You can calcuate confidence intervals for proportions (in R, e.g.
binom::binom.confint shows you a bunch of different methods and approximations).
- In the end of the day, this still stays finite, even if you run large numbers of iterations/repetitions of the cross validation.
- The 2nd source of random uncertainty is model instability. You train a whole lot of "surrogate models" that are assumed to be equvivalent. However, this may not be the case.
- Here, the iterations help: you can measure the variation introduced by instability if you look at the variation of predictions for the same case by different surrogate models (where the test case was of course always excluded from training).
Measuring variance over the iterations of the cross validation will not (or only partly - does anyone know, is there literature about this?) cover the uncertainty due to finite test sample size: if the models are perfectly stable, all surrogate models will yield the same prediction for any given test case. Then there will be no variance between the iterations. But the results are still subject to the finite-test-sample-size type of random uncertainty.
When calculating the confidence interval for such a proportion (using the number of independent samples), the result is usually quite shocking (at least in my field, where we often are happy if tens of cases are available...)
That is also the reason why model comparison based on proportions is not easy at all. You'll want to set this up in a paired way. Otherwise you'd need crazy numbers of test cases.