My dataset is pretty small (120) samples. While doing 10-fold cross validation, should I:
Collect the outputs from each test fold, concatenate them into a vector, and then compute the error on this full vector of predictions (120 samples)?
Or should I instead compute the error on the outputs I get on each fold (with 12 samples per fold), and then get my final error estimate as the average of the 10 fold error estimates?
Are there any scientific papers that argue the differences between these techniques?
Background: Potential Relationship to Macro/Micro scores in multi-label classification:
I think this question may be related to the difference between micro and Macro averages that are often used in a multi-label classification task (e.g. say 5 labels).
In the multi-label setting, micro average scores are computed by making an aggregated contingency table of true positive, false positive, true negative, false negative for all 5 classifier predictions on 120 samples. This contingency table is then used to compute the micro precision, micro recall and micro f-measure. So when we have 120 samples and five classifiers, the micro measures are computed on 600 predictions (120 samples * 5 labels).
When using the Macro variant, one computes the measures (precision, recall, etc.) independently on each label and finally, these measures are averaged.
The idea behind the difference between micro vs Macro estimates may be extended to what can be done in a K-fold setting in a binary classification problem. For 10-fold we can either average over 10 values (Macro measure) or concatenate the 10 experiments and compute the micro measures.
Background - Expanded example:
The following example illustrates the question. Let's say we have 12 test samples and we have 10 folds:
- Fold 1: TP = 4, FP = 0, TN = 8 Precision = 1.0
- Fold 2: TP = 4, FP = 0, TN = 8 Precision = 1.0
- Fold 3: TP = 4, FP = 0, TN = 8 Precision = 1.0
- Fold 4: TP = 0, FP = 12, Precision = 0
- Fold 5 .. Fold 10: All have the same TP = 0, FP = 12 and Precision = 0
where I used the following notation:
TP = # of True Positives, FP = # False Positive, TN = # of True Negatives
The results are:
- Average precision across 10 folds = 3/10 = 0.3
- Precision on the concatenation of the predictions of the 10 folds = TP/TP+FP = 12/12+84 = 0.125
Note that the values 0.3 and 0.125 are very different!