AUC of a cross validation model

Question

Let's say i build a 10X cross-validation model, say with Caret. If i want the AUC of this, is it:

1. The average AUC of the 10 validation samples?
2. Something else?

Cheers!

Answer 1

@cbeleites actually to add on the following statement "But you need to be aware that this is a non-standard procedure and should be explained clearly, and the averages should IMHO ", there is the following paper which states:

"The problem with AUCmerge is that by sorting different folds together, it assumes that the classifier should produce well-calibrated probability estimates. Usually a researcher interested in measuring the quality of the probability estimates will use Brier score or such. By contrast, researchers who measure performance based on AUC typically are unconcerned with calibration or specific threshold values, being only concerned with the classifier’s ability to rank positives ahead of negatives. So, AUCmerge adds a usually unintended requirement on the study: it will downgrade classifiers that rank well if they have poor calibration across folds, as we illustrate in Section 3.2."

https://www.hpl.hp.com/techreports/2009/HPL-2009-359.pdf

Which seems to demonstrate that averaging the AUC should be indeed considered the standard procedure

Answer 2

Summary: I don't think there's anything special with AUC compared to other figures of merit measured by cross validation. For other figures of merit, the cross validation estimate is calculated from the pooled predictions of all surrogate models, so I'd do the same for AUC.

---

This pooling is allwed because one important assumption underlying cross validation is that the surrogate models are equivalent, i.e. the obtained models are stable wrt. exchanging a few training cases.

In addition, you can look at the variation of your figure of merit across the surrogate models and check whether this indicates gross instability. Although iterated/repeated cross validation will allow you to get more straightforward indication of instability.

---

If you want to average per-surrogate-model figures of merit: why not. But you need to be aware that this is a non-standard procedure and should be explained clearly, and the averages should IMHO be weighted to adjust for possibly slightly different numbers of tested cases for each surrogate model.