I have two binary classifiers working on the same dataset. If their ROC AUC values differ by at least 0.05, is it enough to declare that the performance difference between the two classifiers is statistically significant? Because, this is the rule of thumb I have developed over the years to have an idea of if two methods are different enough! If not, then given the score-labels files (and ROC curves,and ROC AUC values) of both methods, how should I proceed?
5
-
5$\begingroup$ Statistical significance typically has to do with a hypothesis test, and you do not appear to be running one. $\endgroup$– DaveNov 21, 2022 at 3:45
-
$\begingroup$ might be related to this question stats.stackexchange.com/questions/26271/… but I care about AUC values rather than just accuracy $\endgroup$– darumaNov 21, 2022 at 3:54
-
4$\begingroup$ Have you estimated the probability to differ by more than 'x' in AUC? Or do q bootstrap on AUC between models and see if they overlap? $\endgroup$– Georg M. GoergNov 21, 2022 at 4:30
-
3$\begingroup$ Are you actually aksing about statistical significance? The question whether the methods are "different enough" sounds more like you are interested in practical significance (i.e. effect size). For interpreting the strength of the difference, the equality of the AUC with the C Statistic might be useful. $\endgroup$– cdalitzNov 21, 2022 at 11:42
-
2$\begingroup$ Sample size always plays a role when determining statistical significance. You've observed some "best guess" estimate of a difference in AUC values, but determining statistical significance will always require a consideration of how much evidence you have to support that guess. $\endgroup$– Nuclear HoagieNov 21, 2022 at 17:13
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Whether a given difference in AUC is statistically significant does not only depend on the absolute difference, but also on the variability of the AUC scores, just the same as for a difference in means between groups (plus distributional assumptions, which people typically handwave away with reference to the CLT).
As Georg M. Goerg writes, I would recommend you bootstrap both models and the difference in AUCs (which will give you an estimate of the variability in this difference). Essentially, you would follow the methodology in R's pROC::roc.test() (see the "Computational Details" section).
answered Nov 21, 2022 at 7:16