I have two binary classifiers working on the same dataset. If their ROC AUC values differ by at least 0.05, is it enough to declare that the performance difference between the two classifiers is statistically significant? Because, this is the rule of thumb I have developed over the years to have an idea of if two methods are different enough! If not, then given the score-labels files (and ROC curves,and ROC AUC values) of both methods, how should I proceed?
Whether a given difference in AUC is statistically significant does not only depend on the absolute difference, but also on the variability of the AUC scores, just the same as for a difference in means between groups (plus distributional assumptions, which people typically handwave away with reference to the CLT).
As Georg M. Goerg writes, I would recommend you bootstrap both models and the difference in AUCs (which will give you an estimate of the variability in this difference). Essentially, you would follow the methodology in R's
pROC::roc.test() (see the "Computational Details" section).