6
$\begingroup$

I was thinking about cross-validation and how it is the most appropriate way to do it...

Let's take the case of binary logistic regression where the goal is to calculate the AUC.

Make the partition of the data using k folds. What is the correct way to get the cross-validated AUC :

1) Train the model using k-1 folds and predict on the kth fold. Calculate the AUC and repeat until all folds served as test set. This will give at the end k AUC values, which we average to get the cross-validated AUC.

2) Train the model using k-1 folds and predict on the kth fold. Save the predictions. Repeat until all folds served as test set. This will give a vector of predictions, one for each subject in the dataset. Calculate the AUC using this vector of predictions and the vector of observed responses.

My intuition and idea of cross-validation suggest that 2) is the correct one...

Improve this question
$\endgroup$
2
$\begingroup$

As Provost explains in 'An Introduction to ROC Analysis', ROC averaging can be simply done by combining the scores from multiple sets $T_1, ..., T_k$ as you suggested in method (2). This is preferred to method (1) since it can be quite hard to average actual ROC curves, since the specificity (x-axis) values of the points are expected to be different. Therefore, you would need to do a lot of interpolation to average the curves. Another advantage is that the resulting curve from method (2) is smoother and approximates the AUC better, as a low number of scores tends to underestimate the AUROC (at least when calculated via the trapezoidal rule).

However, one should note that an advantage of method (1) is that it enables you to estimate the variance of the AUC.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks @bi_scholar ! So my intuition was right... :) $\endgroup$ – GiannisZ Jan 10 '19 at 10:49
  • 2
    $\begingroup$ The whole point of CV is to estimate variance, so only 1) makes sense. CV is intended to provide an initial risk assessment regarding the generalisability of a method. $\endgroup$ – ReneBt Feb 11 '19 at 20:04
  • $\begingroup$ @ReneBt OP asked for the preferred approach to average ROC curves and this is what my answer addresses. Variance estimation is not in the scope of OPs question and thus I merely addressed it with a remark. In context of ROC-averaging, saying that only 1) makes sense is simply false. $\endgroup$ – Scholar Feb 11 '19 at 20:37
  • $\begingroup$ Coming again to this issue for another reason! You both agreed that method (1) is preferable when variance is of interest. So, if you do a k-fold cross-validation, you will end up with a vector of AUCs of length k. Then the variance, i guess, is calculated as normal (variance of the k observations). In case you perform a repeated cross-valdiation, you will end up with m (number of repetitions of the CV) times k AUCs. In that case, is it appropriate to calculate the variance in each repetition, leading to m variances, and then average those m variances ? $\endgroup$ – GiannisZ Nov 12 '19 at 14:55
7
$\begingroup$

The preferred method to do this is (1), rather than (2). For example, based on Forman, G., Scholz, M. & Scholz, M. Apples-to-Apples in Cross-Validation Studies: Pitfalls in Classifier Performance Measurement (2009):

The problem with AUC merge is that by sorting different folds together, it assumes that the classifier should produce well-calibrated probability estimates. Usually a researcher interested in measuring the quality of the probability estimates will use Brier score or such. By contrast, researchers who measure performance based on AUC typically are unconcerned with calibration or specific threshold values, being only concerned with the classifier’s ability to rank positives ahead of negatives. So, AUC merge adds a usually unintended requirement on the study: it will downgrade classifiers that rank well if they have poor calibration across folds, as we illustrate in Section 3.2

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.