# Average vs. combined k-fold cross validation

My dataset is pretty small (120) samples. While doing 10-fold cross validation, should the scores from each test fold combined or should we take the average for the classification task?

For instance, if we combine the prediction, we can calculate precision/recall on 120 predictions. If we take the average, we will have 10 values and an average of that.

Also how do we perform standard deviation or reliability of the measures, should I run 10-fold CV $N$ times and average over that? Will $N=10$ be enough?

Are there any scientific papers that use this technique?

PS: Usage of Macro/Micro scores in multi-label classification and it's relation to k-fold cross valiation

This question may also be related to Micro and Macro averages that are often used in a multi-label classification task ( say one vs. all setting). When there are multiple binary classifiers (say five), micro average scores are computed by making an aggregated contingency table of true positive, false positive, true negative, false negative for all five classifier predictions on 120 samples. This contingency table is then used to compute the micro precision, micro, recall and micro f-measure. So when we have 120 samples and five classifiers, the micro measures are computed on 600 predictions. Macro measure is the average of measure across each classifier. So precision,recall,f-measure for each classifier is measured on 120 predictions and average is computed over five values (from five classifiers). Hence, we get Macro precision, recall and f-measure.

These measures are similar to what can be done in a k-fold setting. For 10 fold we can either average over 10 values (Macro measure) or aggregate over the 10 experiments and compute the Micro measures.

Currently, I'm using the Micro measure to compute each measure of binary classifiers and also have Micro/Macro measure for the whole multi-label classification task involving five classifiers. I was wondering if it was okay to use Micro measure for a binary classification task as this is rare and even in weka and other systems there is only Macro measure for k-fold cross validation.

PS. difference between the combined average (Micro) and average (Macro) in a 10-fold setting Let's say we have 12 test samples in each fold following is the prediction:

T1: TP = 4, FP = 0, TN = 8 Precision = 1.0

T2: TP = 4, FP = 0, TN = 8 Precision = 1.0

T3: TP = 4, FP = 0, TN = 8 Precision = 1.0

T4: TP = 0, FP = 12, Precision = 0

T5..T10 also have TP = 0, FP = 12 and Precision = 0

TP = True Positive, FP = False Positive, TN = True Negative

Average precision over 10 folds = 3/10 = 0.3

Combined precision over 10 folds = TP/TP+FP = 12/12+84 = 0.125

-
CV is not really a great measure of predicting future performance. The variance is just too small. Better to go with bootstrap for validating your model. – user765195 Aug 19 '12 at 20:09
@user765195: could you backup your claim with some citations? – Zach Aug 19 '12 at 20:33
I've been searching but I haven't found any literature regarding the aggregated CV method. It seems to be a more appropriate way to compute the measure as it has less variance. – user13420 Aug 19 '12 at 21:15
@Zach, there's some discussion here, in Harrell's book: tinyurl.com/92fsmuv (look at the last paragraph in page 93 and the first paragraph in page 94.) I'll try to remember other references that are more explicit. – user765195 Aug 19 '12 at 23:42
AFAIK, deciding between out-of-bootstrap and iterated $k$-fold cross validation is not quite that clear. It may depend on the type of data you have and on the interpretation that you want to do. – cbeleites Aug 20 '12 at 9:02

yes, you should run iterations of the whole $k$-fold cross validation procedure:
From that, you can get an idea of the stability of the predictions of your models

• How much do the predictions change if the training data is perturbed by exchanging a few training samples?
• I.e., how much do the predictions of different "surrogate" models vary for the same test sample?

You were asking for scientific papers:

Underestimating variance Ultimately, your data set has finite (n = 120) sample size, regardless of how many iterations of bootstrap or cross validation you do.

• You have (at least) 2 sources of variance in the resampling (cross validation and out of bootstrap) validation results:

• variance due to finite number of (test) sample
• variance due to instability of the predictions of the surrogate models
• If your models are stable, then

• iterations of $k$-fold cross validation were not needed (they don't improve the performance estimate: the average over each run of the cross validation is the same).
• However, the performance estimate is still subject to variance due to the finite number of test samples.
• If your data structure is "simple" (i.e. one single measurement vector for each statistically independent case), you can assume that the test results are the results of a Bernoulli process (coin-throwing) and calculate the finite-test-set variance.
• out-of-bootstrap looks at variance between each surrogate model's predictions. That is possible with the cross validation results as well, but it is uncommon. If you do this, you'll see variance due to finite sample size in addition to the instability. However, keep in mind that some pooling has (usually) taken place already: for cross validation usually $\frac{n}{k}$ results are pooled, and for out-of-bootstrap a varying number of left out samples are pooled.
Which makes me personally prefer the cross validation (for the moment) as it is easier to separate instability from finite test sample sizes.

-
 Also, I'm doing multi-label classification with four classifiers. So I want to look into the Micro and Macro F-measures across the 4 task. I assume the "combined" cross-validation would be even necessary in this case ? Also I'm not certain if the out-of-bootstrap is same as the "combined" CV method I'm mentioning above. There was also some discussion at stats.stackexchange.com/questions/4868/… – user13420 Aug 20 '12 at 14:37 @user13420: neither am I sure what you mean with combined CV... Here's an answer where I wrote down what out-of-bootstrap and cross validation mean to me: stats.stackexchange.com/a/26548/4598 – cbeleites Aug 20 '12 at 16:24 @user13420: Terminology is very different in different fields. Can you update your answer with information what Micro and Macro F-measures are? However, cross validation is a very general technique: it is a scheme for computing model testing results. You can calculate any performance measure that needs as input the reference value for each case and the predicted value for each case. – cbeleites Aug 20 '12 at 16:27 combined CV would mean you collect the prediction of each holdout and compute the measure after all 10 holdouts. So, if I measure precision, recall for a classification task, it would have a single precision,recall as opposed to 10 values and an average (which is the case in usual CV) – user13420 Aug 20 '12 at 18:15 average is not same as the combined average. I've added explanation in the question. – user13420 Aug 20 '12 at 21:30
show 1 more comment