I am using binary classification and at the moment mainly 10-fold crossvalidation. However the result of the crossvalidation is very variable - hence what I was doing recently was to run the 10-fold crossvalidation itself repeatedly, and then average over these classification accuracies (I know that the accuracy of each crossvalidation is already an average of the accuracies for the 10 different test folds). Is this a sensible/common thing to do - and does it make sense to also calculate the variance of these iterations?


Yes! That is very sensible and common procedure. It is often referred to as Repeated cross-validation in literature. Some reference if you can by-pass the paywall is here : Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap by Ji-Hyun Kim.

As you say your classification is producing variable results, please also do report the variance (or standard deviation) in the final results.

  • $\begingroup$ Many many thanks!:) Will read on and possibly ask further questions here @discipulus $\endgroup$
    – Pugl
    Feb 3 '17 at 12:47
  • 2
    $\begingroup$ I don't agree. This is not the same as OP asks. See this article for comparison lirias.kuleuven.be/retrieve/186558. There, the authors warn of using the average of k-fold cross-validation and then repeating it $n$ times to reduce the variance. $\endgroup$
    – JJR
    Nov 30 '20 at 10:09
  • 1
    $\begingroup$ While the variance of the repeated cross-validation is not the variance of estimates of the cross-validation accuracy on new datasets of that size for that problem, they are an indication of the variance due to the partitioning of the data to form the cross-validation folds. If differences in performance of competing models are smaller than the differences due to the formation of the folds, then they are probably not worth getting exited about, so this variance calculation can still useful and worth reporting - depending on the purpose of the CV. $\endgroup$ yesterday
  1. It is a mistake to equate "common" and "sensible".

  2. As the comment by Dikran Marsupial mentions, such a procedure could give a meaningful answer to a question about the stability of cross validation across multiple partitions, but it is a mistake to think that $100$ iterations of $10$-fold cross validation, averaging the performance across the $10$ folds each of the $100$ times, is equivalent to $1000$ separate out-of-sample tests.

Think of it this way. The random variable of interest, $X$, is the distribution of out-of-sample performance scores. We are interested in $Var(X)$. When we do $10$-fold cross validation and average the performance on the $10$ folds, we get $\bar X$, and $Var(\bar X)=\frac{Var(X)}{10}$. By looking at the $100$ values of $\bar X$, we estimate $Var(\bar X)$ instead of $Var(X)$.

The procedure proposed in the OP will underestimate the true variance and mislead us into thinking there is more stability in the out-of-sample score than there really is.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.