1
$\begingroup$

Let's say i build a 10X cross-validation model, say with Caret. If i want the AUC of this, is it:

  1. The average AUC of the 10 validation samples?
  2. Something else?

Cheers!

$\endgroup$
  • $\begingroup$ The average makes sense to me. $\endgroup$ – Yilun Zhang Mar 2 '18 at 19:11
  • $\begingroup$ read here and here $\endgroup$ – phiver Mar 3 '18 at 10:18
2
$\begingroup$

Summary: I don't think there's anything special with AUC compared to other figures of merit measured by cross validation. For other figures of merit, the cross validation estimate is calculated from the pooled predictions of all surrogate models, so I'd do the same for AUC.


This pooling is allwed because one important assumption underlying cross validation is that the surrogate models are equivalent, i.e. the obtained models are stable wrt. exchanging a few training cases.

In addition, you can look at the variation of your figure of merit across the surrogate models and check whether this indicates gross instability. Although iterated/repeated cross validation will allow you to get more straightforward indication of instability.


If you want to average per-surrogate-model figures of merit: why not. But you need to be aware that this is a non-standard procedure and should be explained clearly, and the averages should IMHO be weighted to adjust for possibly slightly different numbers of tested cases for each surrogate model.

$\endgroup$
  • $\begingroup$ Thanks you for answering that. So it's the pooled predictions Vs their outcomes. It makes sense. A 5 fold cross validation would have 5 different testing sets and we group them together and get the AUC. Does the the overall number match the total sample? This answer to this question on how to CV AUC using Caret seems to suggest that. stackoverflow.com/questions/23806556/… $\endgroup$ – khhc Mar 4 '18 at 18:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.