Mann-Whitney significance and multiple hypothesis testing significance

Question

I am performing various comparisons using the Mann-Whitney U test. Because I'm comparing various groups simultaneously, I am also correcting the p-values for multiple hypothesis testing using Bonferroni correction (global significance considered is $0.05$).

Now, consider for instance that I have some small samples, and all the tests need to be corrected for multiple hypothesis testing. In another post (see: Significance level as function of sample size in Mann Whitney) one of the answers states that the comparison of samples with sizes $n1=2$ and $n2=3$ detect differences at a $0.05$ significance level. If I was testing 3 different hypotheses, only $p-value < 0.05/3$ would be statistically significant. But this would mean that larger samples would be required, correct? So, instead of sample sizes of $2$ and $3$ to be able to detect any differences at the $0.05$ level, I would instead need $n1=4$ and $n2=3$ or $n1=2$ and $n2=4$?

I am a bit confused about all this... Could someone please help me out understand how to plug in both the sample size and multiple hypothesis testing? Am I thinking correctly?

Answer 1

A sample size of 2 and 3 in the Mann-Whitney has a minimum achievable significance level of 20% two-tailed, not 5%. Did you halve the alpha-level in the table to get a two-tailed alpha instead of doubling?

To have an achievable significance level of 0.05/3 = 1/60 ($\frac13\times\frac{1}{20}=\frac{1}{60}$) two-tailed you'd need to look up a one-tailed in the table of $\alpha=\frac{1}{120}$.

Let's extend the table, but for space reasons I'm dropping the "1/". So here's the reciprocals of the smallest attainable $\alpha$ (we're now looking for values that are at least 120):

   m:  2    3    4    5    6    7     8
n                                     
2      6   
3     10   20 
4     15   35   70 
5     21   56  126* 252 
6     28   84  210  462  924 
7     36  120* 330  792 1716 3432 
8     45  165  495 1287 3003 6435 12870

The smallest pair of sample sizes that have that small an attainable significance level would be (4,5); alternatively (3,7) also gets there (just), as would (2,14), though that's not in the above table.

If you were doing 6 such comparisons, you'd need a 2-tailed alpha of 0.05/6 $\leq$ 1/120 so you'd in that one-tailed table you'd need $\alpha\leq$ 1/240, or $1/\alpha\geq$ 240 which takes you out to (5,5) or (4,7).

Ten such comparisons pushes you out to (5,6), and so on.

You can always check your calculations by conducting a Wilcoxon-Mann-Whitney test on a pair of non-overlapping samples with the sizes you worked out. For example, in R:

> wilcox.test(1:4,5:9)

        Wilcoxon rank sum test

data:  1:4 and 5:9
W = 0, p-value = 0.01587
alternative hypothesis: true location shift is not equal to 0

Note that 0.01587 is 2/126 (rounded off), just under 0.05/3 = 0.01667, so the values of (4,5) we worked out from the table was correct.

---

However if this is anything other than an intellectual exercise ("Out of curiosity, how low could sample size go?"), I'd be much more concerned about power than achievable significance levels. If you're in planning stages I strongly urge you to consider some power calculation for some meaningful minimum effect size, which will surely increase your required sample sizes above 4-5.

[If you're going to have such tiny sample sizes, why take $\alpha=0.05$ in any case? With such tiny samples your type II error rate may be so high that your type I error rate is almost irrelevant by comparison. There's nothing actually special or useful about 0.05 other than people are used to seeing it; in many cases it's much too high, but if only very small samples are feasible it's quite possibly too low.]