# I have 2 AUCs from the same data but 2 algorithms. How I determine if one of the AUCs is greater in a statistically significant sense

Problem:

I have 200k data samples which are class imblanced (10% positive class, 90% negative class). I split the data in exactly half so my training set is 100k samples and my test set os 100k samples. I train algorithms A and B.

Algorithm A discriminates between the two classes using the test set and it achieves an AUC of AUC_A. Algorithm B, which is an improvement on A, gets AUC we call AUC_B. I want to determine if AUC_B > AUC_A by chance or not (statistically significant).

What is an algorithm to determine this answer? (say we set p<0.001)

• Can you explain more what you mean with "unfair"? If you bootstrap the test set, you would will just get the same answer as without bootstrapping, wouldn't you? Let's say AUC_A < AUC_B. If you bootstrap the test set and you do enough bootstrapping rounds N you will converge to the exact same answer AUC_A < AUC_B. Since the p-value depends on the sample size, you would could just choose N large enough and get an arbitrary small p-value. Right? Do you think the same happens when sampling the training set? So, what is the point of bootstrapping the test set if you already know the answer? – Sören Jun 3 '18 at 16:43