# Problem definition

Suppose I want to test if a classifier is of any use in telling if a person is currently affected by a disease. I have trained my classifier on a training set and now I have its confusion matrix for all probability thresholds of belonging to the positive class. Note that both datasets are very unbalanced because most of the people are actually healthy.

I can then plot the ROC curve: on wikipedia I find that the bisector of the ROC spaces is equivalent to a "random guess".

My classifier in most of the ROC space is above the bisector # Questions

My questions are then:

1. Am I able to reject the null hypothesis (is my classifier of any use)? If not how could that be done properly in a binary classification problem (feel free to provide references if the answer is too big)?
2. I'm feeling I can't because I don't have "error bars" on the ROC curve: if I train several classifier with the same parameters but different train/test splitting would it be sufficient?
3. Is then the null-hypothesis rejection valid only for the probability thresholds where the ROC curve (together with the error bar described in the previous point) is above the ROC space bisector?
• The AUC is a semi-proper scoring rule. If you are open to assessing your probabilistic predictions directly using proper scoring rules, then this thread may be helpful in assessing whether your classifications are better than a population average. Oct 21, 2021 at 10:07
• ROC curves are only appropriate when doing retrospective sampling e.g. case-control designs (to align with the conditioning used for the points on the ROC which condition on the future to predict the past) and you also seem to be wanting to use forced-choice classification when probability estimation should be the goal. And testing a null hypothesis here is unhelpful. Instead look at the variation in the distribution of predicted risks as done here. Oct 21, 2021 at 11:33
– Dave
Oct 21, 2021 at 11:57
• The AUC is intrinsically linked to the Mann-Whitney U-statistic. You can derive a test from that. See stats.stackexchange.com/questions/206911/… Oct 21, 2021 at 12:14
There are other ways to test if your categories have different distributions of the features, but this seems totally legitimate to me. If the model struggles to distinguish between the categories, then you get a low AUC close to $$0.5$$ and a high p-value, as you should. If the features provide valuable information for distinguishing between the categories, model performance will be strong, leading to a high AUC and a small p-value.
For this particular use case, you might need more than just a small improvement over baseline (AUC $$=0.5$$) performance to warrant clinical use. The specific predicted probabilities might be of use there, and you can run tests directly on those (such as a test of a full logistic regression against the intercept-only model nested within it).