Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I have data from 10-fold cross-validation experiment: for each fold I have a predictor and a response variable so I can generate ROC curve and compute area under the ROC curve.
I have a series of three such experiments, so in general I can generate 30 ROC curves. I wonder if anybody has an idea how to average ROC curves over 10 folds for each experiment and then test if differences between three averaged ROC curves are statistically significant.
There is a difference between averaging over AUC and over the curves. Also if you want to do it over the curves then there are a few ways to do it. If you are interested in rate constrained tasks (For example information retrieval with a probability distribution over a time limit, or say classifying customers for a call centre to target in a given time) Then I would recommend this method:
Millard, Louise AC, Meelis Kull, and Peter A. Flach. "Rate-Oriented Point-Wise Confidence Bounds for ROC Curves." Machine Learning and Knowledge Discovery in Databases. Springer Berlin Heidelberg, 2014. 404-421.
https://drive.google.com/file/d/0BzEymYqJrJmhNEdGZWlzaV91d1k/view?usp=sharing
I'm not sure you can sensibly ask about statistically significant differences in this context. From each of the 3 CV procedures you can get an expected AUC from each of the 3 independent data sources. But for the 10 folds within any of those CV procedures the data are not independent, so I you can't get a meaningful variance figure to test significance. You would need to split your data up into truly independent pieces, evaluate the AUC of each (getting replicates for each of the 3 groups), and then use a statistical test.
@OncoStat just answered with something that implies this approach while I was writing up this answer: don't test the differences between the ROCs but rather the area under them. If you use the same folds in each experiment, you then have a classic paired test, in which case you can use a paired difference test, either $t$ or Wilcoxon. If your folds aren't consistent across the experiments, instead do an unpaired test.
Alternatively, here's a slight reframing: instead of averaging, consider the random ROC for each experiment, so that you have three distributions with ten (curve-valued) samples from each. Then you want to know if those distributions differ. Assuming that the folds aren't paired, you can do a maximum mean discrepancy (MMD) test. That requires a kernel between curves; a reasonable first-guess powerful choice might be a Gaussian kernel based on the functional $L_2$ distance, $k(f, g) = \exp\left( - \gamma \lVert f - g \rVert_2^2 \right)$, choosing $\gamma$ to maximize the MMD statistic.
