Reporting variance of the repeated k-fold cross-validation

Question

I have been using repeated k-fold cross validation and been reporting the mean (of the evaluation metric e.g., sensitivity, specificity) computed as the grand mean across the folds of different runs of the cross validation.

However, I am not sure how I should report the variance. I have found many questions here discussing repeated cross-validation, however, none that I am aware of explicitly answers the question of variance in repeated cross validation tests.

I understand that the total variance is due to: 1) instability of model and 2) limited sample size.

It seems that there are 4 different approaches for computing the variance for repeated k-fold cross validation:

1) the variance of the estimated average performance metric (e.g., accuracy) across runs of the cross validation be a valid estimate of the variance?

2) the pooled variance by pooling run-specific variances (which are computed across different folds of a run of cross validation test).

3) to concatenate the classification results from different fold of a cross validation run in a large vector. For instance, if the number of test data in each fold is 10 and I have a 10-fold CV, the resulting vector for a repetition will be of size 100. Now, if I repeat my cross-validation test 10 times, I will have 10 vectors of size 100, each of which containing the classification results from a 10-fold CV run. Now, I would compute the mean and variance as the case of single run CV.

4) I have also read in (equations 2 and 3 in 1) that the variance is the sum of external variance and the expected internal variance. If I understand correctly, the external variance is the variance of repetition-specific average performances, and the internal variance is the variance across different folds of a run of cross validation.

I would greatly appreciate your help and guidance on which variance would be the appropriate one to report for the repeated cross-validation test.

Thanks,

Answer 1

1 and 3 seem to me as invalid since they do not take into account the dependencies between repeated runs. In other words, repeated k-fold runs are more similar to each other than real repetitions of the experiment with independent data.

2 does not take into account the dependencies between folds within the same run.

I do not know about 4.

A potentially relevant (and discouraging) reference is Bengio & Grandvalet, 2004, "No Unbiased Estimator of the Variance of K-Fold Cross-Validation"

Answer 2

I might be wrong (and I am open to change my mind!) but to my understanding, when we say that CV reduces variance we cannot actually see it by looking at the variance across the folds.

The thing is, CV is just a techinque that improves the estimation of the performance of a (one!) model (since in the k-folds we train the same model, simply with different data). Therefore, one round of CV simply gives us one value (the mean across the folds) and we cannot say anything about variance.

As far as I understood, we can introduce variance if we do more experiments. For example, let's consider repeating 10 experiments: in each experiment we run a k-fold-CV and we end up with the final estimation of, say, accuracy. This ultimately gives us 10 different values for accuracy. Now we can compute the variance accross these 10 values, call it $\sigma^2_k$. Personally, I would report this value. Note that we can do the same for the initial scenario without CV. Just compute the final metric for 10 experiments in which you do simply a 70-30 or a 60-40 split!

Now we can do the same thing for other values of k and we can see how this $\sigma^2$ decreases for k=5 or k=10 with respect to the simplest case without CV.