Variance of classification accuracy? I am using binary classification and at the moment mainly 10-fold crossvalidation. However the result of the crossvalidation is very variable - hence what I was doing recently was to run the 10-fold crossvalidation itself repeatedly, and then average over these classification accuracies (I know that the accuracy of each crossvalidation is already an average of the accuracies for the 10 different test folds). Is this a sensible/common thing to do - and does it make sense to also calculate the variance of these iterations?
 A: Yes! That is very sensible and common procedure. It is often referred to as Repeated cross-validation in literature. Some reference if you can by-pass the paywall is here : Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap by Ji-Hyun Kim.
As you say your classification is producing variable results, please also do report the variance (or standard deviation) in the final results.
A: *

*It is a mistake to equate "common" and "sensible".


*As the comment by Dikran Marsupial mentions, such a procedure could give a meaningful answer to a question about the stability of cross validation across multiple partitions, but it is a mistake to think that $100$ iterations of $10$-fold cross validation, averaging the performance across the $10$ folds each of the $100$ times, is equivalent to $1000$ separate out-of-sample tests.
Think of it this way. The random variable of interest, $X$, is the distribution of out-of-sample performance scores. We are interested in $Var(X)$. When we do $10$-fold cross validation and average the performance on the $10$ folds, we get $\bar X$, and $Var(\bar X)=\frac{Var(X)}{10}$. By looking at the $100$ values of $\bar X$, we estimate $Var(\bar X)$ instead of $Var(X)$.
The procedure proposed in the OP will underestimate the true variance and mislead us into thinking there is more stability in the out-of-sample score than there really is.
