# Significance level as function of sample size in Mann Whitney

I was reading this question and answers and this blog post and I was thinking about the minimum significance level that could be achieved by comparing two populations whose medians are statistically different. I would like to know

1. How do you compute the minimum p-value that permit distinguishing the two populations?
2. At a 0.05 significance level, what is the minimum size that each of the populations and that the sum of both populations need to have to permit distinguishing them? From the table at the 2nd link (this) I believe this would be 3 for one of the populations as long as the sum of both was 8. Correct?

I am aware that this will depend on the data. For instance, if there are many ties in two populations, the number of points required are probably more than if there are no ties. Nevertheless, is there something you could say with respect to my two questions?

• Note that the Wilcoxon-Mann-Whitney is not a test for equality of medians* (in spite of countless books in certain application areas repeating this error). $\quad$ * at least not without additional assumptions, which would as easily make it a test for equality of almost any other measure of location. The bogpig link in your question is also incorrect (or at least misleading); the Wilcoxon-Mann-Whitney is not only a test for a location-shift. If you add an assumption (make the alternative a location shift) then it will be, but of itself it is not so restricted. – Glen_b Oct 12 '15 at 21:31
• Thanks! Yes, I understand that one of the assumptions is that the Mann-Whitney test both populations should exhibit similar distributions so that you may take some conclusions about medians of populations. – Sosi Oct 13 '15 at 9:21
• Yes, that's what you can do to test against an alternative of a shift in medians - but with that assumption then you're testing equality (vs a shift) of any other measure of location at the same time (i.e. it's also a test for shift in lower quartiles, or a test for a shift in population means*). $\quad$ * under the weak condition that population means exist. – Glen_b Oct 13 '15 at 9:28
• oh! Then how would you suggest to proceed in this regard? :( I'm so sorry if all of these assumptions seem wrong – Sosi Oct 13 '15 at 9:30
• Nothing changes - apparently you understand the additional assumption in a test of location-shift; but the Wilcoxon-Mann-Whitney test is able to handle a much wider range of alternatives than that. All the information in the answer relates to any of them. [The critical issue is that however you set your test assumptions and alternatives up, under the null you need to have $P(X>Y)=P(Y>X)$, at least in the continuous case] – Glen_b Oct 13 '15 at 9:35

How do you compute the minimum p-value that permit distinguishing the two populations?

Note that p-value of itself doesn't lead to a decision that would "distinguish the two populations"; that's a function of its relationship the significance level. So I will interpret your question to be "What is the minimum attainable p-value at a given pair of sample sizes?"

I'll answer under the assumption of no ties (I may add a little about ties later but it's a more complicated question).

Let's say the two samples are of size $m$ and $n$. Then the total number of ways of allocating the $m+n$ observations to groups of those sizes is ${{m+n} \choose m}=\frac{(m+n)!}{m!n!}$.

The test statistic is equivalent to the sum of the ranks in sample 1.

If all the ranks are distinct (no ties) then there's a single combination (the smallest $m$ ranks) that will minimize the test statistic.

Consequently for a one-tailed test, the smallest attainable p-value is

$$p^*=\frac{1}{{{m+n} \choose m}}=\frac{m!n!}{(m+n)!}$$

and for a two-tailed test, it's twice that.

At a 0.05 significance level, what is the minimum size that each of the populations and that the sum of both populations need to have to permit distinguishing them?

Again assuming no ties, for fixed $m+n$ the smallest attainable one-tailed p-values occur when $m=n$ (or as close as possible to equal if $m+n$ is odd). Let's assume $m\leq n$, consider a table of $p*$:

  p*          m
2      3      4      5      6
2  1/6
n 3  1/10   1/20
4  1/15   1/35  1/70
5  1/21   1/56  1/126  1/252
6  1/28   1/84  1/210  1/462  1/924


For one-tailed tests the smallest combinations of sample sizes that can attain significance at the 5% level are (3,3) and (2,5). Note that exact equality ($p=0.05$) is in the rejection region for a 5% test.

The two-tailed case the p-values are twice as large; to find the cases where the two-tailed p-values can be below 0.05 we need to look for denominators $\geq 40$ in the table above. So (4,4) and (3,5), but not (2,6) (in fact if the smaller sample is size 2, the second sample must be at least size 8).

This table can be computed by hand (I did the first 6 or 7 values that way) but if you use R, this will print the whole table:

MASS:::fractions(outer(2:6,2:6,function(x,y) 1/choose(x+y,x)))


However, if you want to use larger values of $m$ and $n$ than $6$, drop the call to fractions. (If needed as exact fractions, one can more easily just print the denominators in any case and take the "1/" as given for the one-tailed no-ties case.)

Some discussion of ties:

In the case of ties: (i) ties can reduce the total number of combinations; (ii) ties can also increase the number of extreme combinations up from 1 (in a one-tailed test) to some larger number. As a result, ties increase the minimum p-value you can obtain.

Consider the following three sets of ranks (with m=4 and n=5):

  1  2 3 4.5 4.5 6 7 8 9


Note that the most "smallest" combination can happen two ways.