How is it possible that for two sample sets I'm getting a low p-value, but also a low AUC value (just below 0.5)?

To compute the P-value I'm looking at the second outputted value of the function here http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.mannwhitneyu.html

For the AUC I'm using the same function's first outputted value divided by the product of the sample sets lengths.

And here is a boxplot of the two series:

boxplot of the two series

  • 1
    $\begingroup$ Because this question is at least partly tied with a specific software/language which some people potentially able to help you don't use - you should give a data and the results they produce. So that people can try to reproduce the results in the software they know. $\endgroup$ – ttnphns Nov 5 '14 at 12:35
  • $\begingroup$ Are you sure you're getting the right statistic? - you might be calculating the area above the ROC curve. $\endgroup$ – Scortchi - Reinstate Monica Nov 11 '14 at 13:16
  • $\begingroup$ Yeah but it's still close to 0.5 so same same. $\endgroup$ – Uri Nov 12 '14 at 8:09
  • $\begingroup$ @ttnphns I've added a boxplot. I don't really know how to transfer the sample sets. $\endgroup$ – Uri Nov 12 '14 at 10:34
  • $\begingroup$ What sufficiently large sample sizes the area under the receiver operating characteristic curve (AUC) could be "close" to 0.5, yet the one-sided p-value for the Mann-Whitney test "low" (as long as the AUC is greater than 0.5). Another thing to check might be exactly how the statistic's computed - there are some different versions. $\endgroup$ – Scortchi - Reinstate Monica Nov 12 '14 at 10:48

Just looking at the boxplot heavily indicated that @Scortchi is correct with his comment. The number of outliers alone indicate that you have a very high sample size, so a very high power to find differences. This means you have strong evidence for a very small discrimination, which is usually not of high interest (practically speaking).

Mann-Whitney p-values (using the normal approximation) vs AUC for some different sample sizes ($n_1,n_2$):

AUC vs p-value

| cite | improve this answer | |
  • $\begingroup$ If I choose a random subset of 500 samples of each set I see some correlation between AUC and the p-value, but can still get a significant p-value (0.03) and low AUC (0.53) $\endgroup$ – Uri Nov 12 '14 at 12:44
  • $\begingroup$ Could you explain "a very high power to find differences"? $\endgroup$ – Uri Nov 12 '14 at 12:47
  • $\begingroup$ @Uri: There should be no question of a correlation between AUC & the calculated p-value for given sample sizes: An observed AUC $c$ together with sample sizes $n_1$ & $n_2$ determines a p-value of $1-\Phi\left[n_1n_2 \left(c - \frac{1}{2}\right) / \sqrt{\frac{n_1n_2(n_1+n_2+1)}{12}}\right]$ where $\Phi(\cdot)$ is the standard Gaussian distribution function. With both sample sizes at 500, an AUC of a little over 0.53 would give a p-value of 0.03, so your calculation seems correct. $\endgroup$ – Scortchi - Reinstate Monica Nov 12 '14 at 13:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.