# Averaging ROC curves over folds in cross-validation

I have data from 10-fold cross-validation experiment: for each fold I have a predictor and a response variable so I can generate ROC curve and compute area under the ROC curve.

I have a series of three such experiments, so in general I can generate 30 ROC curves. I wonder if anybody has an idea how to average ROC curves over 10 folds for each experiment and then test if differences between three averaged ROC curves are statistically significant.

• First part: From each fold you get pairs of values (sensitivity, specificity), through which the ROC curve for that fold goes. I would just combine all those pairs of values across the 10 folds to get one ROC curve for the "experiment". Commented Jul 22, 2015 at 13:52

@OncoStat just answered with something that implies this approach while I was writing up this answer: don't test the differences between the ROCs but rather the area under them. If you use the same folds in each experiment, you then have a classic paired test, in which case you can use a paired difference test, either $t$ or Wilcoxon. If your folds aren't consistent across the experiments, instead do an unpaired test.
Alternatively, here's a slight reframing: instead of averaging, consider the random ROC for each experiment, so that you have three distributions with ten (curve-valued) samples from each. Then you want to know if those distributions differ. Assuming that the folds aren't paired, you can do a maximum mean discrepancy (MMD) test. That requires a kernel between curves; a reasonable first-guess powerful choice might be a Gaussian kernel based on the functional $L_2$ distance, $k(f, g) = \exp\left( - \gamma \lVert f - g \rVert_2^2 \right)$, choosing $\gamma$ to maximize the MMD statistic.