My dataset is pretty small (120) samples. While doing 10-fold cross validation, should the scores from each test fold combined or should we take the average for the classification task?
For instance, if we combine the prediction, we can calculate precision/recall on 120 predictions. If we take the average, we will have 10 values and an average of that.
Also how do we perform standard deviation or reliability of the measures, should I run 10-fold CV $N$ times and average over that? Will $N=10$ be enough?
Are there any scientific papers that use this technique?
PS: Usage of Macro/Micro scores in multi-label classification and it's relation to k-fold cross valiation
This question may also be related to Micro and Macro averages that are often used in a multi-label classification task ( say one vs. all setting). When there are multiple binary classifiers (say five), micro average scores are computed by making an aggregated contingency table of true positive, false positive, true negative, false negative for all five classifier predictions on 120 samples. This contingency table is then used to compute the micro precision, micro, recall and micro f-measure. So when we have 120 samples and five classifiers, the micro measures are computed on 600 predictions. Macro measure is the average of measure across each classifier. So precision,recall,f-measure for each classifier is measured on 120 predictions and average is computed over five values (from five classifiers). Hence, we get Macro precision, recall and f-measure.
These measures are similar to what can be done in a k-fold setting. For 10 fold we can either average over 10 values (Macro measure) or aggregate over the 10 experiments and compute the Micro measures.
Currently, I'm using the Micro measure to compute each measure of binary classifiers and also have Micro/Macro measure for the whole multi-label classification task involving five classifiers. I was wondering if it was okay to use Micro measure for a binary classification task as this is rare and even in weka and other systems there is only Macro measure for k-fold cross validation.
PS. difference between the combined average (Micro) and average (Macro) in a 10-fold setting Let's say we have 12 test samples in each fold following is the prediction:
T1: TP = 4, FP = 0, TN = 8 Precision = 1.0
T2: TP = 4, FP = 0, TN = 8 Precision = 1.0
T3: TP = 4, FP = 0, TN = 8 Precision = 1.0
T4: TP = 0, FP = 12, Precision = 0
T5..T10 also have TP = 0, FP = 12 and Precision = 0
TP = True Positive, FP = False Positive, TN = True Negative
Average precision over 10 folds = 3/10 = 0.3
Combined precision over 10 folds = TP/TP+FP = 12/12+84 = 0.125