For cross-validation of a logistic regression classifier, it seems to me that there are a number of different approaches to calculating each performance metric:
- The performance metric is calculated separately in each left out fold, repeated across k folds, then averaged.
- The probabilities are calculated in each left out fold, repeated across k folds, concatenated into a long vector and a single performance metric calculated.
- Same as 2 but instead of concatenating the probabilities, store the indexes and average the predicted probabilities for each outcome instance. Calculate a single performance metric from the averaged probabilities.
This question has been asked before with respect to AUCs here. This question considered the first two options above and there was no consensus as to the correct method. The accepted answer suggested the first method was correct but the answer with the most upvotes suggested the second method and referenced a paper, "Apples-to-Apples in Cross-Validation Studies: Pitfalls in Classifier Performance Measurement", to support this.
Another question asks the same question and the answer suggests there is merit to both the first and second approaches.
This question is also relevant.
The issue is also addressed with reference to AUCs in Statistical Evaluation of Diagnostic Performance: Topics in ROC Analysis p204:
the machine learning community often uses other strategies to calculate the cross-validated AUC. For example, Bradley pointed out that some averaged AUCs from ROC curves correspond to each partition and others aggregated the outputs of all folds first, producing one ROC and calculating its AUC
Altogether there does not appear to be a consensus.
Another motivation for asking the question again is that I cannot find any reference that has considered how to calculate performance metrics other than the AUC, like the calibration slope (beta) or net benefit by decision curve analysis, by cross-validation. Does anyone have any advice?