Let's say i build a 10X cross-validation model, say with Caret. If i want the AUC of this, is it:
- The average AUC of the 10 validation samples?
- Something else?
Summary: I don't think there's anything special with AUC compared to other figures of merit measured by cross validation. For other figures of merit, the cross validation estimate is calculated from the pooled predictions of all surrogate models, so I'd do the same for AUC.
This pooling is allwed because one important assumption underlying cross validation is that the surrogate models are equivalent, i.e. the obtained models are stable wrt. exchanging a few training cases.
In addition, you can look at the variation of your figure of merit across the surrogate models and check whether this indicates gross instability. Although iterated/repeated cross validation will allow you to get more straightforward indication of instability.
If you want to average per-surrogate-model figures of merit: why not. But you need to be aware that this is a non-standard procedure and should be explained clearly, and the averages should IMHO be weighted to adjust for possibly slightly different numbers of tested cases for each surrogate model.