# Can I use a Mann-Whitney U test to compare 2 small groups, with unequal amounts of data, non-normal distributions, and unequal variance?

I want to compare data between 2 independent groups to determine whether there is a significant difference between the groups. 1 group has 8 data points, and the other has 9. Separately, I would like to compare data between 2 more independent groups, in which 1 group has 8 data points and the other has 43.

According to a Shapiro-Wilk test, my groups do not reflect normality. According to an F-test, my groups do not reflect equal variance. Some sources claim that equal variance is an assumption of the Mann-Whitney U test, while others claim that equal variance is not necessary to use the test. Some sources claim that a Brunner-Munzel test would be more appropriate as an alternative for the Mann-Whitney U test, citing that it would account for unequal variance. What should I do?

test to compare 2 small groups,

Sure, it's made to work just fine as a small-sample test. It also works perfectly well as a large sample test; some books and websites are actively misleading on this point, some directly claiming it's not a large sample test (indeed I found another as I was writing this answer).

As long as the samples are not so small that you can't attain reasonable significance levels (this is not usually a big problem; with 8 and 9 observations you can get a significance level of 4.64% as well as numerous smaller ones -- as long as you don't have ties; if there are ties the calculation I did won't work and a new calculation would be required).

Of course, small samples typically means low power, and all the problems that come with it, but the test still does what it claims to do.

with unequal amounts of data,

Ditto, it's meant to work for the case of non-identical sample sizes.

non-normal distributions,

It doesn't particularly care what the distribution is (normal or otherwise), as long as under the null, the distribution is the same (see additional discussion below); if you don't have continuous distributions there are some wrinkles to worry about but they don't stop you using the test.

and unequal variance?

What you need is that under the null, the random variables representing the sample values are exchangeable. Without getting into the technical details of that, it's a weaker requirement than assuming the set of random variables representing all the values are independent and identically distributed (i.i.d.), which people are often content to assume when there's no effect (you assume this -- and more besides -- when you use an ordinary two sample t-test, for example).

Which is to say, you could certainly use the test if it would be reasonable to assume the samples would be i.i.d. when $$H_0$$ were true -- meaning that it would have the correct significance level (as long as you choose from the set of available significance levels at your sample sizes).

Naturally, for it to have good power, you also need some conditions on the way it behaves when $$H_0$$ is false (it's not just that absolutely anything goes), but that doesn't have to include having the same variance.

Indeed, since it's a rank based test, consider what happens under a monotonic but nonlinear transformation of the data. Clearly this cannot change the value of the test statistic, nor p-values from it; the information in the test statistic about the hypotheses is unchanged, because the collection of ranks is unaltered. Yet it can certainly change the relative size of the spreads under the alternative, so if they're equal-variance under one transformation they generally won't be under other typical transformations.

So, for example, if I had a set of times and took their reciprocal (yielding rates or speeds), and imagine that the population spreads of those speeds are essentially identical. Then spreads for the original times won't be the same if the mean speeds differed, but the test doesn't care one iota whether you look at speeds or times. A test that's invariant to monotonic transformation can't rely on identical spreads.

[To clarify, inverting the values will alter the direction of an effect but with a two sample test this doesn't matter, as long as we keep in mind that larger times corresponds to smaller speeds when writing our conclusions. Even for a one tailed test we need merely flip directions in hypotheses when we invert the measurement; this is not consequential.]

You will see many books and web-pages claim that the Wilcoxon-Mann-Whitney necessarily assumes equal variances.

The test assuredly does not require that in general, and it is certainly not required in the samples; what is needed is what is necessary to get exchangeability under the null. In short, they're just wrong when they claim you need it and they're also wrong when they tell you to test for it in your sample, it's neither helpful nor without consequences for you.

You still have a question to ponder (the reasonableness of exchangeability under $$H_0$$), but that's a much weaker requirement than equal spreads under both hypotheses.

Now, clearly the data cannot be used to judge whether the population assumption is true under the null, since you don't know that the null is true - if you could tell that, what would be the point of testing at all?

Further, the data also should not be used to decide what test you will use on the same data, since that definitely impacts the properties of the tests you'll be using (p-values would no longer have their required properties under the null, for starters). In short, as a general rule don't test your assumptions on the data you want to use in the original test, it screws up your hypothesis tests.

So you both can't, and shouldn't, conclude that assumptions that you want to make under the null actually hold by looking at the sample itself. You will need some other basis to come to that conclusion.

One example of the sort of reasoning that might be used in some situation (not necessarily your case, about which not much was said, but this is just an example):

"It's anticipated that either the new treatment will tend to shift the distribution to be typically larger, or that the suggested pathway is not involved, in which case it should have no effect on the distribution"

That would lead to an immediate assumption of identical distribution under the null, but does not require that the variance remain the same under the alternative.

This inequality of variance under the alternative is fine, the Wilcoxon-Mann-Whitney behaves as it should.

However, note that if the variance changes very rapidly under the alternative (i.e. the spread changes a lot with even very small changes to $$P(X), then the power properties may tend to be poor.

In short there's nothing in what you've mentioned so far that's problematic.

If you don't think that the spreads would be approximately equal when $$H_0$$ was true (i.e. when $$P(XY)$$) then it would be safer to use the test you mentioned, but outside that, I see nothing of concern.

If I did doubt that equal variance under $$H_0$$ condition held and if the samples were at the same time a bit larger than you have, I might be inclined to construct a form of bootstrap test, but with very small samples maintaining close to the required significance level can be harder under a bootstrap test, so I might well consider the Brunner Munzel test in that circumstance.

Yes, for unequal sample sizes and non-normal data. But be careful using this test for small samples or noticeably different variances.

Power is very poor if either sample has fewer then about five observations. In the example below, you might think it is clear that samples x1 and x2 come from populations with different medians, the test (as implemented in R) finds no difference. There are only $${6 \choose 3} = 20$$ ways to put the six observations into two groups of three, and two of those ways provide such extreme 'separation'. So, there is one chance in ten of getting such different samples.

x1 = c(1,2,3)
x2 = c(98,99,100)
wilcox.test(x1, x2)

Wilcoxon rank sum test

data:  x1 and x2
W = 0, p-value = 0.1
alternative hypothesis:
true location shift is not equal to 0


Moreover, If you need to interpret the Mann-Whitney-Wilcoxon rank sum test as a test of unequal population medians, then the two samples should be of similar shape and that implies nearly-equal variances.

Consider the following fictitious data, where variances are far from equal.

set.seed(325)
x1 = rnorm(7, 50, 10)
x2 = rnorm(20, 51, 2)

median(x1)
[1] 44.33793
median(x2)
[1] 51.56419


When variances differ, this test is often for stochastic dominance rather than for difference in medians. Here the test fails to find any difference at the 5% level.

wilcox.test (x1, x2)

Wilcoxon rank sum test

data:  x1 and x2
W = 38, p-value = 0.08127
alternative hypothesis:
true location shift is not equal to 0


Not surprisingly, a plot of the empirical CDFs of the two samples is not so easy to interpret. Mostly, x2 (blue) dominates x1, plotting mainly below and to the right, but domination is not clear-cut.

plot(ecdf(x1), col="brown")
lines(ecdf(x2), col="blue")


Because data are normal, one could have used a Welch 2-sample t test (which does not require equal variances), but it also fails to find a difference at the 5% level. [Roughly speaking, the small DF can be regarded as a 'penalty' for unequal sample variances.]

t.test(x1, x2)

Welch Two Sample t-test

data:  x1 and x2
t = -2.1647, df = 6.264, p-value = 0.07169
alternative hypothesis:
true difference in means is not equal to 0
95 percent confidence interval:
-13.1340230   0.7371156
sample estimates:
mean of x mean of y
45.00702  51.20548