I am testing two predictive models, Each of them in 10 experiments.

I therefore have 10 ROC curves (with AUC) for first experiment and 10 ROC curves for second experiment.

How to compare these series and see if difference between them is statistically significant ?

How to get confidence intervals / P value ?

  • $\begingroup$ Does this help with the confidence intervals stats.stackexchange.com/questions/204068/…? $\endgroup$
    – mdewey
    Apr 28 '17 at 15:18
  • $\begingroup$ There is a r package that can do confidence intervals and p value for a given one series for one model. However I am trying to compare series of experiments for 2 different models.... $\endgroup$
    – suprvisr
    Apr 28 '17 at 15:19
  • $\begingroup$ Do you have 20 independent experiments each giving you an AUC? $\endgroup$
    – mdewey
    Apr 28 '17 at 15:38
  • $\begingroup$ No I have 10 for one model and 10 for another model - and need to compare these... $\endgroup$
    – suprvisr
    Apr 28 '17 at 15:39
  • $\begingroup$ There is an R package metamisc but it does not allow for a moderator. You could e-mail the authors to see if he has any plans to include that or, since it uses metafor internally, you could hack it to include moderators. $\endgroup$
    – mdewey
    Apr 28 '17 at 15:45

Browse other questions tagged or ask your own question.