Averaging ROC curves over folds in cross-validation I have data from 10-fold cross-validation experiment: for each fold I have a predictor and a response variable so I can generate ROC curve and compute area under the ROC curve.
I have a series of three such experiments, so in general I can generate 30 ROC curves. I wonder if anybody has an idea how to average ROC curves over 10 folds for each experiment and then test if differences between three averaged ROC curves are statistically significant.
 A: I'm not sure you can sensibly ask about statistically significant differences in this context. From each of the 3 CV procedures you can get an expected AUC from each of the 3 independent data sources. But for the 10 folds within any of those CV procedures the data are not independent, so I you can't get a meaningful variance figure to test significance.  You would need to split your data up into truly independent pieces, evaluate the AUC of each (getting replicates for each of the 3 groups), and then use a statistical test.
A: @OncoStat just answered with something that implies this approach while I was writing up this answer: don't test the differences between the ROCs but rather the area under them. If you use the same folds in each experiment, you then have a classic paired test, in which case you can use a paired difference test, either $t$ or Wilcoxon. If your folds aren't consistent across the experiments, instead do an unpaired test.
Alternatively, here's a slight reframing: instead of averaging, consider the random ROC for each experiment, so that you have three distributions with ten (curve-valued) samples from each. Then you want to know if those distributions differ. Assuming that the folds aren't paired, you can do a maximum mean discrepancy (MMD) test. That requires a kernel between curves; a reasonable first-guess powerful choice might be a Gaussian kernel based on the functional $L_2$ distance, $k(f, g) = \exp\left( - \gamma \lVert f - g \rVert_2^2 \right)$, choosing $\gamma$ to maximize the MMD statistic. 
A: There is a difference between averaging over AUC and over the curves. 
Also if you want to do it over the curves then there are a few ways to do it. If you are interested in rate constrained tasks (For example information retrieval with a probability distribution over a time limit, or say classifying customers for a call centre to target in a given time)  Then I would recommend this method: 
Millard, Louise AC, Meelis Kull, and Peter A. Flach. "Rate-Oriented Point-Wise Confidence Bounds for ROC Curves." Machine Learning and Knowledge Discovery in Databases. Springer Berlin Heidelberg, 2014. 404-421.
https://drive.google.com/file/d/0BzEymYqJrJmhNEdGZWlzaV91d1k/view?usp=sharing 
