Let's say i build a 10X cross-validation model, say with Caret. If i want the AUC of this, is it:
- The average AUC of the 10 validation samples?
- Something else?
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.Sign up to join this community
@cbeleites actually to add on the following statement "But you need to be aware that this is a non-standard procedure and should be explained clearly, and the averages should IMHO ", there is the following paper which states:
"The problem with AUCmerge is that by sorting different folds together, it assumes that the classifier should produce well-calibrated probability estimates. Usually a researcher interested in measuring the quality of the probability estimates will use Brier score or such. By contrast, researchers who measure performance based on AUC typically are unconcerned with calibration or specific threshold values, being only concerned with the classifier’s ability to rank positives ahead of negatives. So, AUCmerge adds a usually unintended requirement on the study: it will downgrade classifiers that rank well if they have poor calibration across folds, as we illustrate in Section 3.2."
Which seems to demonstrate that averaging the AUC should be indeed considered the standard procedure
Summary: I don't think there's anything special with AUC compared to other figures of merit measured by cross validation. For other figures of merit, the cross validation estimate is calculated from the pooled predictions of all surrogate models, so I'd do the same for AUC.
This pooling is allwed because one important assumption underlying cross validation is that the surrogate models are equivalent, i.e. the obtained models are stable wrt. exchanging a few training cases.
In addition, you can look at the variation of your figure of merit across the surrogate models and check whether this indicates gross instability. Although iterated/repeated cross validation will allow you to get more straightforward indication of instability.
If you want to average per-surrogate-model figures of merit: why not. But you need to be aware that this is a non-standard procedure and should be explained clearly, and the averages should IMHO be weighted to adjust for possibly slightly different numbers of tested cases for each surrogate model.