# Is a 0.05 absolute difference in AUC values enough to declare statistically significant difference?

I have two binary classifiers working on the same dataset. If their ROC AUC values differ by at least 0.05, is it enough to declare that the performance difference between the two classifiers is statistically significant? Because, this is the rule of thumb I have developed over the years to have an idea of if two methods are different enough! If not, then given the score-labels files (and ROC curves,and ROC AUC values) of both methods, how should I proceed?

• Statistical significance typically has to do with a hypothesis test, and you do not appear to be running one.
– Dave
Nov 21, 2022 at 3:45
• might be related to this question stats.stackexchange.com/questions/26271/… but I care about AUC values rather than just accuracy Nov 21, 2022 at 3:54
• Have you estimated the probability to differ by more than 'x' in AUC? Or do q bootstrap on AUC between models and see if they overlap? Nov 21, 2022 at 4:30
• Are you actually aksing about statistical significance? The question whether the methods are "different enough" sounds more like you are interested in practical significance (i.e. effect size). For interpreting the strength of the difference, the equality of the AUC with the C Statistic might be useful. Nov 21, 2022 at 11:42
• Sample size always plays a role when determining statistical significance. You've observed some "best guess" estimate of a difference in AUC values, but determining statistical significance will always require a consideration of how much evidence you have to support that guess. Nov 21, 2022 at 17:13