Problem definition

Suppose I want to test if a classifier is of any use in telling if a person is currently affected by a disease. I have trained my classifier on a training set and now I have its confusion matrix for all probability thresholds of belonging to the positive class. Note that both datasets are very unbalanced because most of the people are actually healthy.

I can then plot the ROC curve: on wikipedia I find that the bisector of the ROC spaces is equivalent to a "random guess".

My classifier in most of the ROC space is above the bisector enter image description here


My questions are then:

  1. Am I able to reject the null hypothesis (is my classifier of any use)? If not how could that be done properly in a binary classification problem (feel free to provide references if the answer is too big)?
  2. I'm feeling I can't because I don't have "error bars" on the ROC curve: if I train several classifier with the same parameters but different train/test splitting would it be sufficient?
  3. Is then the null-hypothesis rejection valid only for the probability thresholds where the ROC curve (together with the error bar described in the previous point) is above the ROC space bisector?
  • 1
    $\begingroup$ The AUC is a semi-proper scoring rule. If you are open to assessing your probabilistic predictions directly using proper scoring rules, then this thread may be helpful in assessing whether your classifications are better than a population average. $\endgroup$ Oct 21, 2021 at 10:07
  • $\begingroup$ ROC curves are only appropriate when doing retrospective sampling e.g. case-control designs (to align with the conditioning used for the points on the ROC which condition on the future to predict the past) and you also seem to be wanting to use forced-choice classification when probability estimation should be the goal. And testing a null hypothesis here is unhelpful. Instead look at the variation in the distribution of predicted risks as done here. $\endgroup$ Oct 21, 2021 at 11:33
  • $\begingroup$ @FrankHarrell That link is broken: “Page not found”. $\endgroup$
    – Dave
    Oct 21, 2021 at 11:57
  • 2
    $\begingroup$ The AUC is intrinsically linked to the Mann-Whitney U-statistic. You can derive a test from that. See stats.stackexchange.com/questions/206911/… $\endgroup$
    – Firebug
    Oct 21, 2021 at 12:14
  • $\begingroup$ Sorry about the incorrect link. Please use fharrell.com/post/addvalue $\endgroup$ Oct 21, 2021 at 12:17

1 Answer 1


The ROC curve measures how well the model can distinguish between the two categories: the higher the AUC score, the better the ability to distinguish (at least a bit loosely speaking). This comes from the fact that the ROCAUC is related to running a two-sample Wilcoxon Mann-Whitney U test on the continuous predictions made by your model, such as predicted probabilities or log-odds of a logistic regression.

You can get the test statistic for the hypothesis test from knowing the AUC and the number of members of each category. Then you can get a viable p-value.

There are other ways to test if your categories have different distributions of the features, but this seems totally legitimate to me. If the model struggles to distinguish between the categories, then you get a low AUC close to $0.5$ and a high p-value, as you should. If the features provide valuable information for distinguishing between the categories, model performance will be strong, leading to a high AUC and a small p-value.

For this particular use case, you might need more than just a small improvement over baseline (AUC $=0.5$) performance to warrant clinical use. The specific predicted probabilities might be of use there, and you can run tests directly on those (such as a test of a full logistic regression against the intercept-only model nested within it).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.