What is the null hypothesis in the Mann-Whitney test?

Question

Let $X_1$ be a random value from distribution 1 and let $X_2$ be a random value from distribution 2. I thought that the null hypothesis for the Mann-Whitney test was $P(X_1 < X_2) = P(X_2 < X_1)$.

If I run simulations of the Mann-Whitney test on data from normal distributions with equal means and equal variances, with $\alpha=0.05$, I get Type I error rates which are very close to 0.05. However, if I make the variances unequal (but leave the means equal), the proportion of simulations in which the null hypothesis is rejected becomes larger than 0.05, which I didn't expect, since $P(X_1 < X_2) = P(X_2 < X_1)$ still holds. This happens when I use wilcox.test in R, regardless of whether I have exact=TRUE, exact=FALSE, correct=TRUE, or exact=FALSE, correct=FALSE.

Is the null hypothesis something different from what I've written above, or is it just that the test is inaccurate in terms of Type I error if the variances are unequal?

Answer 1

From Hollander & Wolfe pp 106-7,

Let $F$ be the distribution function corresponding to population 1 and $G$ be the distribution function corresponding to population 2. The null hypothesis is: $H_O: F(t)=G(t)$ for every $t$. The null hypothesis asserts that the $X$ variable and the $Y$ variable have the same probability distribution, but the common distribution is not specified.

Strictly speaking this describes the Wilcoxon test, but $U=W-\frac{n(n+1)}{2}$, so they're equivalent.