Summary: Whatever you do in order to calculate confidence intervals based on repeated CV, you need to take into account that there are several different sources of uncertainty.
Long version: Let me add my 2 ct with regard to repeated cross validation:
Repeated cross validation allows you to separate 2 sources of variance uncertainty in the test results.
- variance due to model instability, i.e. variance in the predictions for the same case by different surrogate models (i.e. exchange a few training cases in the surrogate models' training sets) and
- variance due to the finite (limited) number of independent test cases
Now typically cross validation assumes that the models are stable, i.e. variance 1. is negligible. This assumption you can easily check. If you find it non-negligible, you'd typically go back to training and try to stabilize your models before doing anything else.
Variance 2 depends heavily on the total number of independent cases = the total number of independent cases tested in each run of the cross validation (and is usually much worse if you insist on classification rather than staying with metric scores).
I'm pointing this out because repetitions of the cross validation can help estimating and reducing variance 1, but will not mitigate variance 2 - but under the standard assumptions for cross validation, variance 2 should be dominating.
I suspect that this the underlying cause for VanWinckelen's finding that "Repeated cross-validation should not be assumed to
give much more precise estimates of a model’s predic-
If, instead of characterizing the model you get using the data at hand, you try to find out whether one or the other algorithm would be better for similar applications, you have more unknown sources of uncertainty, see Bengio and Grandvalet: No unbiased estimator of the variance of k-fold cross-validation.
Disclaimer: I cannot say much on the area under those curves as for my applications the curves are often skewed and I rather need to take into account pairs of figures of merit such as sens and spec or PPV and NPV (or scoring rules that are analogous to them).