Main question:
I have an imbalanced binary labeled dataset (6% positive labels) and two different methods of training a predictive model for binary classification (e.g. Tree Model vs. Neural Network), let's call them method/model A and B in this post. I am interested in comparing the performance of these models measured in AUC (AUC-ROC and AUC-PR). I want to make a solid comparison and show that the difference is significant (or not) when I say for example that model A has a better AUC than model B, and not just report a simple difference after one training cycle, as one could then not know if the difference could be attributed to random elements (caused by the random data split).
I have come across multiple options, and am not sure how to implement them or how I can combine them. I will list some concepts I have found, and make a suggestion of how to test the performance difference of which I am not sure whether it is unnecessarily complicated or even valid.

Cross Validation (CV)
As far as I know, CV is used for two reasons: to get a better understanding of the (out of sample) performance (generalizability) of a model, the error estimation, and second to find the best hyperparameters. When it is used for both at the same time, nested cross-validation (NCV) is required. See the Wikipedia Page on this topic. They distinguish between two variants, but I believe that the cases that they describe come down to exactly the same. The idea is for this k*l-fold cross-validation is as follows. We have an outer loop of let's say five iterations. Within each iteration we split the dataset into a development (dev) set and a test set. Within each iteration we perform another loop of let's say five iterations, where we split the dev set into five different folds of training + validation set. This give in total 25 iterations, where each iteration has a unique fold of train/test/val split. Please see the image below for an overview. The last iteration of the outer loop also shows the inner loop consisting of five iterations as well. Image taken from this paper.


There are two options of testing and calculating a metric within each outer fold:

  1. We can use the five folds over the dev set (inner loop iterations) to find the optimal hyper parameters, and once found, retrain once on the entire dev set (so train + val instead of just train) and report the performance once applied on the test set. This will produce one metric (let's say AUC) per out iteration, so will give 5 AUCs in total after the entire NCV.
  2. Each inner iteration, we use the train set to train, using the dev set to find the right hyperparameters and stopping criterion, and then immediately apply the trained model (on only the train set) on the test set. This will produce one AUC per inner iteration, so 25 AUCs in total. Note that even within each outer iteration, the 5 AUCs might be based on different hyperparameters, so it is hard to report what the best hyperparameters are.

I believe I have to go for the second option, because when using my Neural Network I will have to use the dev set not only for the hyperparameters, but also as an early stopping indicator to prevent overfitting. I would not know how to stop the training using only a train and a test set, because the loss on the train set will only go down, and we can of course not look at the performance on the test set prematurely as this would leak information and give a biased indication of the out of sample performance. Applying this second option will then give me 25 AUCs. I was wondering whether I can check for the pairwise difference in AUCs between technique A and B (each pair will have trained and tested on the same data, so I believe I can compare them). I could then maybe take the mean of these 25 differences (let's call them deltas), and calculate the standard deviation of these 25 means (to be used as error bars), and see whether a difference of zero falls within this range of +/- 1 std.

I came across the option of bootstrapping the test set to get confidence intervals. One method, the percentile method, looks at the middle 95% percent of calculated metrics on resampled test sets. One could then take for example 1000 resampled test sets, calculate for method A and B the AUC, get the pair-wise difference, and see whether the 0 difference falls within this middle 95%:

Other options

  • Paper by DeLong about comparing ROC-curves, with >18000 citations on Google Scholar.
  • Use Paired Permutation test.
  • Use the standard error of the Wilcoxon statistic, a method mentioned in a paper by Andrew (1997) with >7000 citations on Scholar. It comes down to: enter image description here

Suggestion and questions
For a good balance between simplicity and robustness, I suggest using the second option of the nested cross validation. I will then obtain 25 AUC differences. A question I have then is:

  • Once I have these 25 AUC differences, is it correct that I need to take +/- 2*std (1.96 to be exact) and check whether the value 0 falls in this range, to conclude whether there is a significant difference? Sometimes you see error bars in plots that are +/- 1 std only, while I have read that for the 95% confidence interval, you need to do +/- 2 standard deviations.
  • Would combining this option with the bootstrapping add anything? I would then obtain 25000 AUC differences, and could check for the middle 95% like I showed in the bootstrapping picture.
  • What about the other options I mentioned?

I got kind of lost in all the options. I hope someone can clear this up for me, and let me know if the plan I have makes sense and is valid. Thanks!

  • $\begingroup$ AUROC does not contain the needed information so that differences in AUC provide reasonable power for detecting differences in model performance. Differences in AUC is synonymous with taking differences in two WIlcoxon statistics, which is never done. AUROC does not reward extreme predictions that are “correct” enough. For sensitive measures see this. $\endgroup$ Nov 8 at 12:38

1 Answer 1


This paper, specifically, section "Useful analytic forms for error bars in modeling", subsection "Area under the (ROC) curve (AUC)", has a good overview on comparing ROC/AUC for different models. There is also a section on Bootstrapping error bars.

I'm not a statistician, so I can't comment on the other approaches you suggested. Be aware that CV results are not amenable to t-test, for instance, because the data used is not iid. My last suggestion is: don't try to reinvent the wheel (you would need to demonstrate the validity of any non-published approach).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.