5
$\begingroup$

I am using the Wilcoxon Rank-Sum/Mann-Whitney U test to compare metrics between two groups. In almost all cases, I get huge values for the statistic and only p-values around zero. In one case I got reasonable values for p around 0.22, which raises the question if the other cases are actually around zero. The number of values in the distributions I compare ranges between 200k and 800k depending on the metric and can differ between the two groups.

The null hypothesis is that both groups are from the same distribution.

  • Is it possible to get p-values around zero with these tests and what does that mean?
  • Is there a cause which could explain this behavior?

Please comment if you need more information on my setup.

Edit 1:

The statistic values seem to be reasonable as they seem to be around the expected mean of 1/2 the product of the size of the two groups.

Edit 2:

It seems that I have very few very high outliers but they should not be an issue with the chosen test as described here: Do we need to worry about outliers when using rank-based tests?.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Please can you further motivate your problem by e.g. plotting the two distributions of responses that form your metrics, one for each group? The test statistic for M-W has a mean of around 1/2 the product of sample sizes in both groups, and if the distributions are sufficiently different, the p-value will be small. $\endgroup$
    – B.Liu
    Aug 22, 2021 at 10:33
  • 1
    $\begingroup$ Your null hypothesis is incorrect: the rank sum tests $\text{H}_{0}\text{: }P(X_{A} > X_{B}) = 0.5$ with $\text{H}_{\text{A}}\text{: }P(X_{A} > X_{B}) \ne 0.5$. If you can support the addition stringent added assumptions—(1) the distributions have the same shape, and (2) have the same dispersion—then you can interpret the null as mean equality and as median equality, with the test providing evidence of location shift. Wilcoxon's original article bears a read. :) $\endgroup$
    – Alexis
    Aug 22, 2021 at 16:45
  • $\begingroup$ @Alexis Interesting, honestly I got the hypothesis from the scipy docs and didn't think much about it (docs.scipy.org/doc/scipy/reference/generated/…). $\endgroup$
    – marvinsxtr
    Aug 22, 2021 at 16:59
  • $\begingroup$ @Alexis I read this article on the topic, suggesting that the null hypothesis remains the same in both cases (different or same shape, i.e. detecting equal distributions or changes in median) and only the alternative hypothesis changes. Did I miss something here? $\endgroup$
    – marvinsxtr
    Aug 22, 2021 at 17:54
  • 2
    $\begingroup$ @marvinsxtr You did: the null hypothesis is, as Wilcoxon, and Mann and Whitney wrote, a test for evidence of whether one group is "stochastically larger" than the other, exactly as I expressed in my first comment. If the distributions have different shapes or different dispersions, then it is possible to correctly reject the null I wrote above without a corresponding difference in means, or in medians, and vice versa. See Mann & Whitney also. $\endgroup$
    – Alexis
    Aug 22, 2021 at 19:19

1 Answer 1

Reset to default
11
$\begingroup$

Nothing out of the ordinary is going on from the sound of it.

In almost all cases, I get huge values for the statistic

Have you looked at the range of possible values for the statistic?

For the usual form of the U-statistic, it can take values between $0$ and $mn$ where $m$ and $n$ are the two sample sizes.

If you divide the statistic by $mn$, you get $\frac{U}{mn}$, which is the proportion (rather than the count) of cases in which a value from one sample exceeds a value from the other, which takes values between $0$ and $1$. The null case corresponds to an expected proportion of $\frac12$ (with standard error $\sqrt{\frac{m+n+1}{12mn}}$).

Alternatively, you could look at a z-score, which you may find somewhat more intuitive than the raw test statistic.

Is it possible to get p-values around zero with these tests and what does that mean?

Certainly. Unless your sample sizes are very small, extremely small p-values are possible.

For a one-tailed test the p-value may be as small as $\frac{m!\, n!}{(m+n)!}$ and twice that for a two-tailed test. For example with small sample sizes of $m=n=10$, you could see a two-tailed p-value as small as $1/92378$ or about $0.000011$ and the smallest available p-values decrease very rapidly as sample sizes increase. Doubling both sample sizes to $m=n=20$ reduces the smallest possible p-value by a factor of about $746000$, to $1.45\times 10^{-11}$.

Is there a cause which could explain this behavior?

A small to moderate effect size with large samples or a large effect size with smaller samples can both do it.

Improve this answer
$\endgroup$
9
  • 3
    $\begingroup$ It's salutary to phrase WMW as estimating the probability that values in one group are greater than values in the other. It's common that this is close to 0.5 (whether slightly above or slightly below depends on labelling only). It's an unfortunate detour lasting 70+ years to put more emphasis on the test than on an estimate of this quantity. $\endgroup$
    – Nick Cox
    Aug 22, 2021 at 12:17
  • 3
    $\begingroup$ @NickCox Thanks for that. There's also the immediate fact that U is a count of 'successes' divided by a total number of attempts (It's not binomial, however) that invites a direct 'sample proportion estimating a population proportion` interpretation. Mann and Whitney's original paper talks directly about the probability of an observation from one sample being smaller than an observation from the other under the null (which they identify as being $\frac12$). Indeed, this proportion/probability identification arguably goes all the way ... ctd $\endgroup$
    – Glen_b
    Aug 22, 2021 at 22:48
  • 3
    $\begingroup$ ctd... back to Deuchler (1914); he construes the test as a set of pairwise comparisons between groups, and writes the statistic as the proportion of (A>B) - proportion of (A<B) (i.e. as the difference in counts divided by the total count). That he thought of this in terms of proportions is further evidenced by his use of the binomial in trying to derive the variance of the statistic (albeit incorrectly). See William H. Kruskal (1957) 'Historical Notes on the Wilcoxon Unpaired Two-Sample Test', Journal of the American Statistical Association, 52:279, 356-360 $\endgroup$
    – Glen_b
    Aug 22, 2021 at 22:54
  • 3
    $\begingroup$ Returning to Mann and Whitney, when considering consistency in section 5 they explicitly write $P(x_i>y_j)$ in discussing the expectation of the statistic under the alternative, and then identify $E(U)$ as being $mn$ times this probability; and constructing binary indicator variable for this event ($x_i>y_j$). They seem to be aware of the interpretation. $\endgroup$
    – Glen_b
    Aug 22, 2021 at 22:59
  • 3
    $\begingroup$ Then there's recognizing that the Mann-Whitney U is $mn$ times a U-statistic (for which Hoeffding, 1948 is the main reference, but Lehmann, 1951 ('Consistency and Unbiasedness of Certain Nonparametric Tests,' The Annals of Mathematical Statistics, 22(2), 165-179) discusses the case of two-sample statistics and covers the Mann-Whitney statistic, where the connection to P(Y>X) is explicit. However, all this aside, NickCox's reference will probably be the one you want. $\endgroup$
    – Glen_b
    Aug 22, 2021 at 23:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.