# Averaging ROC curves over folds in cross-validation

I have data from 10-fold cross-validation experiment: for each fold I have a predictor and a response variable so I can generate ROC curve and compute area under the ROC curve.

I have a series of three such experiments, so in general I can generate 30 ROC curves. I wonder if anybody has an idea how to average ROC curves over 10 folds for each experiment and then test if differences between three averaged ROC curves are statistically significant.

• First part: From each fold you get pairs of values (sensitivity, specificity), through which the ROC curve for that fold goes. I would just combine all those pairs of values across the 10 folds to get one ROC curve for the "experiment". Jul 22, 2015 at 13:52

There is a difference between averaging over AUC and over the curves. Also if you want to do it over the curves then there are a few ways to do it. If you are interested in rate constrained tasks (For example information retrieval with a probability distribution over a time limit, or say classifying customers for a call centre to target in a given time) Then I would recommend this method:

Millard, Louise AC, Meelis Kull, and Peter A. Flach. "Rate-Oriented Point-Wise Confidence Bounds for ROC Curves." Machine Learning and Knowledge Discovery in Databases. Springer Berlin Heidelberg, 2014. 404-421.

I'm not sure you can sensibly ask about statistically significant differences in this context. From each of the 3 CV procedures you can get an expected AUC from each of the 3 independent data sources. But for the 10 folds within any of those CV procedures the data are not independent, so I you can't get a meaningful variance figure to test significance. You would need to split your data up into truly independent pieces, evaluate the AUC of each (getting replicates for each of the 3 groups), and then use a statistical test.

@OncoStat just answered with something that implies this approach while I was writing up this answer: don't test the differences between the ROCs but rather the area under them. If you use the same folds in each experiment, you then have a classic paired test, in which case you can use a paired difference test, either $t$ or Wilcoxon. If your folds aren't consistent across the experiments, instead do an unpaired test.

Alternatively, here's a slight reframing: instead of averaging, consider the random ROC for each experiment, so that you have three distributions with ten (curve-valued) samples from each. Then you want to know if those distributions differ. Assuming that the folds aren't paired, you can do a maximum mean discrepancy (MMD) test. That requires a kernel between curves; a reasonable first-guess powerful choice might be a Gaussian kernel based on the functional $L_2$ distance, $k(f, g) = \exp\left( - \gamma \lVert f - g \rVert_2^2 \right)$, choosing $\gamma$ to maximize the MMD statistic.