Significance level as function of sample size in Mann Whitney I was reading this question and answers and this blog post and I was thinking about the minimum significance level that could be achieved by comparing two populations whose medians are statistically different. I would like to know 


*

*How do you compute the minimum p-value that permit distinguishing the two populations?

*At a 0.05 significance level, what is the minimum size that each of the populations and that the sum of both populations need to have to permit distinguishing them? From the table at the 2nd link (this) I believe this would be 3 for one of the populations as long as the sum of both was 8. Correct?


I am aware that this will depend on the data. For instance, if there are many ties in two populations, the number of points required are probably more than if there are no ties. Nevertheless, is there something you could say with respect to my two questions?
 A: 
How do you compute the minimum p-value that permit distinguishing the two populations?

Note that p-value of itself doesn't lead to a decision that would "distinguish the two populations"; that's a function of its relationship the significance level. So I will interpret your question to be "What is the minimum attainable p-value at a given pair of sample sizes?"
I'll answer under the assumption of no ties (I may add a little about ties later but it's a more complicated question).
Let's say the two samples are of size $m$ and $n$. Then the total number of ways of allocating the $m+n$ observations to groups of those sizes is ${{m+n} \choose m}=\frac{(m+n)!}{m!n!}$.
The test statistic is equivalent to the sum of the ranks in sample 1.
If all the ranks are distinct (no ties) then there's a single combination (the smallest $m$ ranks) that will minimize the test statistic.
Consequently for a one-tailed test, the smallest attainable p-value is 
$$p^*=\frac{1}{{{m+n} \choose m}}=\frac{m!n!}{(m+n)!}$$
and for a two-tailed test, it's twice that.

At a 0.05 significance level, what is the minimum size that each of the populations and that the sum of both populations need to have to permit distinguishing them?

Again assuming no ties, for fixed $m+n$ the smallest attainable one-tailed p-values occur when $m=n$ (or as close as possible to equal if $m+n$ is odd). Let's assume $m\leq n$, consider a table of $p*$:
  p*          m
       2      3      4      5      6
  2  1/6   
n 3  1/10   1/20
  4  1/15   1/35  1/70
  5  1/21   1/56  1/126  1/252
  6  1/28   1/84  1/210  1/462  1/924

For one-tailed tests the smallest combinations of sample sizes that can attain significance at the 5% level are (3,3) and (2,5). Note that exact equality ($p=0.05$) is in the rejection region for a 5% test.
The two-tailed case the p-values are twice as large; to find the cases where the two-tailed p-values can be below 0.05 we need to look for denominators $\geq 40$ in the table above. So (4,4) and (3,5), but not (2,6) (in fact if the smaller sample is size 2, the second sample must be at least size 8).
This table can be computed by hand (I did the first 6 or 7 values that way) but if you use R, this will print the whole table:
MASS:::fractions(outer(2:6,2:6,function(x,y) 1/choose(x+y,x)))

However, if you want to use larger values of $m$ and $n$ than $6$, drop the call to fractions. (If needed as exact fractions, one can more easily just print the denominators in any case and take the "1/" as given for the one-tailed no-ties case.)

Some discussion of ties:
In the case of ties: (i) ties can reduce the total number of combinations; (ii) ties can also increase the number of extreme combinations up from 1 (in a one-tailed test) to some larger number. As a result, ties increase the minimum p-value you can obtain. 
Consider the following three sets of ranks (with m=4 and n=5):
  1  2 3 4.5 4.5 6 7 8 9 

Note that the most "smallest" combination can happen two ways.
