1
$\begingroup$

Problem:

I have 200k data samples which are class imblanced (10% positive class, 90% negative class). I split the data in exactly half so my training set is 100k samples and my test set os 100k samples. I train algorithms A and B.

Algorithm A discriminates between the two classes using the test set and it achieves an AUC of AUC_A. Algorithm B, which is an improvement on A, gets AUC we call AUC_B. I want to determine if AUC_B > AUC_A by chance or not (statistically significant).

What is an algorithm to determine this answer? (say we set p<0.001)

$\endgroup$
1
$\begingroup$

If you want a p-value you need many values. How about bootstraping the training set? Then you build N models. For each model you make a prediction on the test set. Here the test set stays the same. You get two distribution for the AUC’s. These you can check with standard staticial tests. E.g. Welch’s test.

Also have a look at this

$\endgroup$
  • $\begingroup$ Bootstrapping using the training set is unfair. You would have to do it on the test set. $\endgroup$ – Avedis Jun 3 '18 at 11:01
  • $\begingroup$ Can you explain more what you mean with "unfair"? If you bootstrap the test set, you would will just get the same answer as without bootstrapping, wouldn't you? Let's say AUC_A < AUC_B. If you bootstrap the test set and you do enough bootstrapping rounds N you will converge to the exact same answer AUC_A < AUC_B. Since the p-value depends on the sample size, you would could just choose N large enough and get an arbitrary small p-value. Right? Do you think the same happens when sampling the training set? So, what is the point of bootstrapping the test set if you already know the answer? $\endgroup$ – Sören Jun 3 '18 at 16:43
1
$\begingroup$

I used the wilcoxon ranked sum test to solve the problem. It is designed to determine if one scoring yields better results than another when the samples are paired. Additionally, I used the Bonferonni correction since I did multiple comparisons.

I computed many AUCs by using bootstrapping and fed the results to the wilcoxon ranked sum test.

https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ranksums.html

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.