I would like to find if there is a significant difference between two ROC-curves. I've found the roc.test in the pROC package. However, I cannot seem to find any information on how this test is actually performed. Can anyone explain it to me or refer to a webpage that has information on this?

From the documentation for roc.test(), I've found that the used test is DeLong's test (based on the article 'Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach'). However, I don't quite understand the article.

My ROC-curves have been found by comparing two logistic regressions, where one has a subset of attributes of the other seeing as I want to examine whether these attributes are useful. The probabilities are found using 5-fold cross-validation.

  • 1
    $\begingroup$ You can see in the documentation under References that the original paper is DeLong et al. (1988) "Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach". Biometrics 44, pp. 837-45. I don't remember all the details, but the gist of it is that you can use a version of the Mann-Whitney U-test to compare the AUCs. $\endgroup$ – gung Jun 18 '14 at 22:10
  • $\begingroup$ Thank you! I don't know Mann-Whitney U-test. Is it safe to use DeLong's test for my purpose? It seems good that it is non-parametric. $\endgroup$ – pir Jun 19 '14 at 14:14
  • $\begingroup$ The description of the package can be found here: "pROC: an open-source package for R and S+ to analyze and compare ROC curves" Xavier Robin, Natacha Turck, Alexandre Hainard, Natalia Tiberti, Frédérique Lisacek, Jean-Charles Sanchez and Markus Müller, BMC Bioinformatics 2011, 12:77, doi:10.1186/1471-2105-12-77 $\endgroup$ – Viktor Jul 27 '15 at 4:22
  • $\begingroup$ How would cross validation help with "finding" the probabilities? Cross-validation is used to study the performance of a model, not to fit the model. $\endgroup$ – Frank Harrell Aug 26 '15 at 10:57
  • 1
    $\begingroup$ @FrankHarrell I think what he meant is that he performed a 5-fold CV to generate scores for every sample which he then used to construct the ROC curve. $\endgroup$ – bi_scholar Dec 17 '18 at 0:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.