Is the ROC curve sufficient for rejecting the null hypotesis in binary classifications?

Question

Problem definition

Suppose I want to test if a classifier is of any use in telling if a person is currently affected by a disease. I have trained my classifier on a training set and now I have its confusion matrix for all probability thresholds of belonging to the positive class. Note that both datasets are very unbalanced because most of the people are actually healthy.

I can then plot the ROC curve: on wikipedia I find that the bisector of the ROC spaces is equivalent to a "random guess".

My classifier in most of the ROC space is above the bisector

Questions

My questions are then:

1. Am I able to reject the null hypothesis (is my classifier of any use)? If not how could that be done properly in a binary classification problem (feel free to provide references if the answer is too big)?
2. I'm feeling I can't because I don't have "error bars" on the ROC curve: if I train several classifier with the same parameters but different train/test splitting would it be sufficient?
3. Is then the null-hypothesis rejection valid only for the probability thresholds where the ROC curve (together with the error bar described in the previous point) is above the ROC space bisector?

Answer 1

The ROC curve measures how well the model can distinguish between the two categories: the higher the AUC score, the better the ability to distinguish (at least a bit loosely speaking). This comes from the fact that the ROCAUC is related to running a two-sample Wilcoxon Mann-Whitney U test on the continuous predictions made by your model, such as predicted probabilities or log-odds of a logistic regression.

You can get the test statistic for the hypothesis test from knowing the AUC and the number of members of each category. Then you can get a viable p-value.

There are other ways to test if your categories have different distributions of the features, but this seems totally legitimate to me. If the model struggles to distinguish between the categories, then you get a low AUC close to $0.5$ and a high p-value, as you should. If the features provide valuable information for distinguishing between the categories, model performance will be strong, leading to a high AUC and a small p-value.

For this particular use case, you might need more than just a small improvement over baseline (AUC $=0.5$) performance to warrant clinical use. The specific predicted probabilities might be of use there, and you can run tests directly on those (such as a test of a full logistic regression against the intercept-only model nested within it).