Let's say i build a 10X cross-validation model, say with Caret. If i want the AUC of this, is it:

  1. The average AUC of the 10 validation samples?
  2. Something else?


  • $\begingroup$ The average makes sense to me. $\endgroup$
    – TYZ
    Mar 2, 2018 at 19:11
  • $\begingroup$ read here and here $\endgroup$
    – phiver
    Mar 3, 2018 at 10:18

2 Answers 2


@cbeleites actually to add on the following statement "But you need to be aware that this is a non-standard procedure and should be explained clearly, and the averages should IMHO ", there is the following paper which states:

"The problem with AUCmerge is that by sorting different folds together, it assumes that the classifier should produce well-calibrated probability estimates. Usually a researcher interested in measuring the quality of the probability estimates will use Brier score or such. By contrast, researchers who measure performance based on AUC typically are unconcerned with calibration or specific threshold values, being only concerned with the classifier’s ability to rank positives ahead of negatives. So, AUCmerge adds a usually unintended requirement on the study: it will downgrade classifiers that rank well if they have poor calibration across folds, as we illustrate in Section 3.2."


Which seems to demonstrate that averaging the AUC should be indeed considered the standard procedure


Summary: I don't think there's anything special with AUC compared to other figures of merit measured by cross validation. For other figures of merit, the cross validation estimate is calculated from the pooled predictions of all surrogate models, so I'd do the same for AUC.

This pooling is allwed because one important assumption underlying cross validation is that the surrogate models are equivalent, i.e. the obtained models are stable wrt. exchanging a few training cases.

In addition, you can look at the variation of your figure of merit across the surrogate models and check whether this indicates gross instability. Although iterated/repeated cross validation will allow you to get more straightforward indication of instability.

If you want to average per-surrogate-model figures of merit: why not. But you need to be aware that this is a non-standard procedure and should be explained clearly, and the averages should IMHO be weighted to adjust for possibly slightly different numbers of tested cases for each surrogate model.

  • $\begingroup$ Thanks you for answering that. So it's the pooled predictions Vs their outcomes. It makes sense. A 5 fold cross validation would have 5 different testing sets and we group them together and get the AUC. Does the the overall number match the total sample? This answer to this question on how to CV AUC using Caret seems to suggest that. stackoverflow.com/questions/23806556/… $\endgroup$
    – khhc
    Mar 4, 2018 at 18:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.