3
$\begingroup$

I've two ROC derived from caret and I'd like to test if the relative curves are statistically different:

library(MLeval)

## Unique ML group of library's toy data
levels(as.factor(predsc$Group))

## Comparing two models
res <- evalm(list(fit1,fit2),gnames=c('ranger','gbm'), rlinethick=0.8, fsize=8, plots='r')

enter image description here

How I can verify if these ROCs are statistically different? In past I used roc.test but I don't figure out how to use it with this MLeval objects.

Improve this question
$\endgroup$
2
  • $\begingroup$ What do you want to know, if the curves themselves are different or if the AUCs are different? $\endgroup$
    – Dave
    Commented Oct 27, 2020 at 10:43
  • $\begingroup$ @Dave a need curve comparison p-value. roc.test $\endgroup$
    – Borexino
    Commented Oct 27, 2020 at 10:52

1 Answer 1

Reset to default
2
$\begingroup$

Concordance probability (AUROC; c-index) are not appropriate for comparing two models, because of lack of statistical power. See fharrell.com/post/addvalue for measures that are sensitive enough to be used for such comparisons.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ This is the right answer and where I was going with my comment. However, I can imagine a scenario where a boss/reviewer/tyrant insists on using ROC curves and demands to know if the differences between two are statistically significant. $\endgroup$
    – Dave
    Commented Oct 27, 2020 at 11:09
  • $\begingroup$ I know this happens but you have to stick to principles and avoid putting yourself in a situation where uninformed people rule the day. Bosses can be replaced; there is a shortage of statisticians/data scientists so we have the luxury of not being stuck. But there are rare occasions where doing a supplementary subservient suboptimal analysis that accompanies the appropriate analysis is helpful. $\endgroup$
    – Frank Harrell
    Commented Oct 27, 2020 at 11:17
  • $\begingroup$ The scenario is referring to a reviewer. How I can solve this elegantly? $\endgroup$
    – Borexino
    Commented Oct 27, 2020 at 11:58
  • $\begingroup$ I would say that the test for differences in concordance probabilities (AUROCs) lacks statistical power because it does not reward extreme predictions that are right, being only a function of ranks of predicted values. Tests should be based on proper accuracy scores, one of which is the log likelihood in a statistical model. Differences in c-indexes are equivalent to differences in Wilcoxon statistics, which are never used. I.e. it's comparing how A and B compare to how B and C compare using the Wilcoxon test. Instead we need a head-to-head comparison A vs C. $\endgroup$
    – Frank Harrell
    Commented Oct 27, 2020 at 12:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.