5
$\begingroup$

I am rather confused about the Mann Whitney test, many statements I read state it tests for distribution equality between two populations and some state it tests for means/median/central tendency only. I run some tests and it shows it only tests for central tendency, not shape. Many books state distribution equality (pdf), why? Can you please explain.

------Distribution equality statements-------

  • Sheldon Ross' book Suppose that one is considering two different methods of production in determining whether the two methods result in statistically identical items. To attack this problem let X1,...,Xn, Y1,...,Ym denote samples of the measurable values of items by method 1 and method 2. If we let F and G, both assumed to be continuous, denote the distribution functions of the two samples, respectively, then the hypothesis we wish to test is H0:F=G. One procedure for testing H0 is the Mann-Whitney test. His statement implies pdf equality, right.

  • Some Caltech notes Now suppose we have two samples. We want to know whether they could have been drawn from the same population, or from different populations, and, if the latter, whether they differ in some predicted direction. Again assume we know nothing about probability distributions, so that we need non-parametric tests. Mann-Whitney (Wilcoxon) U test. There are two samples, A (m members) and B (n members); H0 is that A and B are from the same distribution or have the same parent population. Coming from the same population implies same pdfs.

  • Wikipedia This test can be used to investigate whether two independent samples were selected from populations having the same distribution.

  • Nonparametric Statistical Tests The null hypothesis is H0: θ = 0; that is, there is no difference at all between the distribution functions F and G. But when I use F=N(0,10) and G=U(-3,3) and do the test, the p-value is very high. They can't be more different except E(F)=E(G) and symmetric.

-----Mean/median equality statements-------

  • ArticleThe Mann–Whitney U-test can be used when the aim is to show a difference between two groups in the value of an ordinal, interval or ratio variable. It is the non-parametric version of the t-test. many others like that.
  • Test results
pkg load statistics #octave package
x = normrnd(0, 1, [1,100]); #100 N(0,1)
y1 = normrnd(0, 3, [1,100]); #100 N(0,3)
y2 = normrnd(0, 20, [1, 100]); #100 N(0,20)
y3 = unifrnd(-5, 5, [1,100]); #100 U(-5,5)
[p, ks] = kolmogorov_smirnov_test(y1, "norm", 0, 1) #KS test if y1==N(0,1)
p = 0.000002; #y of N(0,3) not equal to N(0,1)
[p, z] = u_test(x, y1); #Mann-Whitney of x~N(0,1) vs y~N(0,3)
p = 0.52; #null accepted 
[p, z] = u_test(x, y2); #Mann-Whitney of x~N(0,1) vs y~N(0,20)
p = 0.32; #null accepted
[p, z] u_test(x, y3); #Mann-Whitney of x~N(0,1) vs y~U(-5,5)
p = 0.15; #null accepted
#Apparently, Mann-Whitney doesn't test pdf equality

-------Confusing---------

  • Nonparametric Statistical Methods, 3rd Edition I don't understand how its H0: E(Y)-E(X) = 0 = no-shift, can be deduced from (4.2) which seems to suggest pdf equality (equal higher moments) except the shift.
  • Article The test can detect differences in shape and spread as well as just differences in medians. Differences in population medians are often accompanied by equally important differences in shape. really??how??...confused.

After-thoughts

It seems many notes teach MW in a duck-typing way in which MW is introduced as a duck because if we only focus on key behaviours of a duck (quack=pdf, swim=shape), MW does appear like a duck (location-shift test). Most of the times, a duck and donald duck don't behave too markedly different, so such a MW description seems fine and easy to understand; but when donald duck dominates a duck whilst still quacking like a duck, MW can show significance, baffling unsuspecting students. It is not the students' fault, but a pedagogical mistake by claiming donald duck is a duck without clarifying he can be un-duck at times.

Also, my feeling is that in parametric hypothesis testing, tests are introduced with their purpose framed in $H_0$, making the $H_1$ implicit. Many authors move on to nonparametric testing without first highlighting differences in getting the test-statistics probabilities (permutating X Y samples under $H_0$), so students continue to differentiate tests by looking at $H_0$.

Like we are taught to use t-test for $H_0:\mu_x = k $ or $H_0: \mu_x = \mu_y$ and F-test for $H_0: \sigma_x^2 = \sigma_y^2$, with $H_1: \mu_x \ne \mu_y$ and $H_1: \sigma_x^2 \ne \sigma_y^2 $ implicit; on the other hand, we need to be explicit about what we test in $H_1$ as $H_0: F=G$ is trivially true for all tests of a permutation nature. So when instead of seeing $H_0: F=G$ and automatically thinking of $H_1: F \ne G$ so it is a K-S test, we should rather pay attention to the $H_1$ in deciding what's under analysis ($F\ne G, F>G $) and pick a test (KS, MW) accordingly.

$\endgroup$
  • 1
    $\begingroup$ Welcome to CV, ABC Analytics. $\endgroup$ – Alexis Aug 1 at 17:55
  • 1
    $\begingroup$ Regarding the edit citing the Nonparametric Statistical Methods book page, read that page carefully. It states that One model that is useful to describe such alternatives is the translation model, also called the location-shift model. The location-shift model is: $\forall t : G(t) = F(t-\Delta)$. This agrees with what Alexis was saying: the location-shift model makes the further assumption that G and F are identically distributed, except for a translation. Under this assumption, the Mann Whitney then becomes a suitable test to check whether that translation is significantly larger than 0. $\endgroup$ – Tasos Papastylianou Aug 3 at 20:43
  • 1
    $\begingroup$ Regarding the cited Article in the edit. I think the point of that article is that one should not immediately interpret a significant Mann Whitney test as denoting a location shift effect, since one could also get a significant Mann Whitney result with a scaling effect (which could also result in a higher median, but for completely different reasons). This conceptual distinction would be important in a clinical trial, for instance, if the experimental group simply happened to be more spread out, rather than shifted due to a consistent effect form the drug in question. $\endgroup$ – Tasos Papastylianou Aug 3 at 20:55
  • $\begingroup$ u_test and wilcox.test don't mention "stochastic dominance" but R says it tests "location shift" without mentioning assumption of distribution-equality. Simply saying MWW is a location-shift test (although not entirely accurate) seems to be the most common description, and easier to understand without needing to imagine higher moments. $\endgroup$ – ABC Analytics Aug 6 at 1:20
  • 1
    $\begingroup$ Of probable interest: stats.stackexchange.com/questions/471486/… I like Bruce’s beta distribution example, because it shows that Wilcoxon can be unusually sensitive to strong differences other than location, even when the mean and median are the same. The gist is that we might prefer Wilcoxon as an alternative to t-testing, accepting the fact that, whatever advantages Wilcoxon has, it has a drawback in possibly being sensitive to differences other than those we with to investigate. $\endgroup$ – Dave Aug 6 at 1:50
1
$\begingroup$

It is informative to see exactly what the Mann-Whitney test does. For two samples $X = \{x_1, \dots, x_m \}$ and $Y=\{y_1, \dots, y_n\}$, under the assumptions that

  • Observations in $X$ are iid
  • Observations in $Y$ are iid
  • The samples $X$ and $Y$ are mutually independent.
  • The respective populations from which $X$ and $Y$ were sampled are continuous.

then, the U statistic is defined as:

$$ U = \sum_{i=1}^m \sum_{j=1}^n bool(x_i < y_j )$$

It should be reasonably intuitive to see that if X and Y represent the same distributions (i.e. the null hypothesis), then the expected value of $U$ would $mn/2$, since you could expect values below a certain rank to occur as often for $X$ as for $Y$. So you can think of the Mann Whitney test as checking to what extent the statistic $U$ deviates from this expected value.

If this intuition isn't clear, then think of the first rank (i.e. the leftmost rarest value in each sample). If $X$ and $Y$ were drawn from the same distribution, you would have no reason to expect that the rarest value in $X$ would be less than $Y$ more than 50% of the time, otherwise this would make you think that actually $X$ has a heavier tail than $Y$. You can extend this logic for the 2nd rarest value, 3rd, and so forth.

Similarly, if you drew the same number of observations, say $K$, you could almost think of the ranks as $K$ "common bins" with fuzzy boundaries. If $X$ and $Y$ came from the same population, you might expect each rank to occupy roughly the same space, and there's no reason to think that the $x_k $observation in that bin would be to the right of $y_k$ more than 50% of the time.

However, if $x_k$ at a particular "bin" $k$ was to the right of $y_k$ more often than not, then this denotes that there is a systematic "shift". This is what makes Mann-Whitney a good test for detecting 'shift' in distributions that are assumed to be relatively similar except for a possible shift due to a treatment effect.

Now consider the $X \sim \mathcal N(0,1)$ vs $Y \sim \mathcal N(0,2)$ scenario. Assume $K=1000$ samples in each case. You would expect that for the most part, given the same rank, negative values in Y, would tend to be to the left of X more or less all the time. Whereas, positive values in Y, would tend to be to the right of X more or less all the time. Therefore in this particular scenario, even though the distributions are completely different, it happens that half the time X is less likely to be larger than Y, and half the time it is more likely. Therefore you'd expect the U statistic to be very close to the expected value $K^2/2$, and therefore unlikely to be significant.

In other words, it may be a reasonable test to compare two samples in a general "goodness of fit" sense in some specific circumstances, but it is important to be familiar with the situations where it would not. The example above is one such case.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ it seems that some authors define MWW in its generality as H0: F=G, while others explicitly state based on assumptions such as F=G besides location or F and G have same shapes/spreads, then MWW tests medians. The u_test() of Octave only tests medians and I have only read procedures of the latter. So are there versions without those assumptions, of which can indeed test more than medians? $\endgroup$ – ABC Analytics Aug 5 at 15:00
  • 1
    $\begingroup$ @ABCAnalytics I doubt there are different "versions". It's just that the output may have different "interpretations", based on the assumptions made. At the end of the day, the MU test tests a very specific thing: whether $p(X<Y) = 0.5$. What a finding of $p(X<Y) \neq 0.5$ ends up meaning in practice depends on the assumptions made for the problem at hand. $\endgroup$ – Tasos Papastylianou Aug 5 at 15:04
  • $\begingroup$ I read some more hand-holding materials such as this and this, which explicitly state the assumptions then introduce the procedure as a median test. Sheldon Ross doesn't explicit state the assumption but introduces the same test as H0: F=G, which caused my confusion. I still don't see if the same-but-location assumption is forced upon, does that mean we should only use MWW when location-shift? or we can use MWW even we know F!=G ? $\endgroup$ – ABC Analytics Aug 5 at 15:28
  • 1
    $\begingroup$ @ABCAnalytics I think you're getting a bit too hung up on the "H0: F=G" statement. In some sense, the H0 is defined based on its H1 of interest. If H1 is phrased in terms of a location shift, assuming equal shapes/spreads (whether explicitly, or implicitly as in Hollander-Wolfe-Chicken), then H0 states that "There is not shift", and therefore, in other words, given the equal shapes/spreads assumption, F=G. For a different H1, like the "test for different spreads given positive distributions and zero origin" pubmed Article, your H1 is different, but H0 is F=G given those assumptions $\endgroup$ – Tasos Papastylianou Aug 5 at 15:41
  • 1
    $\begingroup$ I have to say MW is commonly included but not well introduced in many texts with its multitude of names, confusing descriptions, and different U calculations, while at its essence it is not really complicated. $\endgroup$ – ABC Analytics Aug 6 at 11:24
5
$\begingroup$

Neither

The Mann-Whitney(-Wilcoxon) $U$ test is typically a test of $\text{H}_{0}\text{: }P(X_{A} > X_{B}) = 0.5$, rejected in favor of $\text{H}_{\text{A}}\text{: }P(X_{A} > X_{B}) \ne 0.5$. In plain language: the probability that a randomly selected observation from group $\text{A}$ is greater than a randomly selected observation from group $\text{B}$ is one half (i.e. even odds). This could be interpreted as a test for (0$^\text{th}$-order) stochastic dominance (i.e. the "stochastically larger than" in the title of the seminal paper).

I write 'typically', because there are both one-sided, and negativist (i.e. there is some difference greater than $\delta$) hypotheses for which $U$ forms the basis of the test statistic.

The (frequent) interpretation of the $U$ test as a test for median difference, for mean difference, or for location shift (pick yer interpretation) results from the two additional (stringent) assumptions:

  1. The distributions of group $\text{A}$ and group $\text{B}$ have identical shapes.

  2. The distributions of group $\text{A}$ and group $\text{B}$ have identical variances.

On a personal note, I feel that adding these requisites sharply curtails the generality of the $U$ test's application by tying it to distributional assumptions beyond the (within group) i.i.d. assumption.



References
Mann, H. B., & Whitney, D. R. (1947). On A Test Of Whether One Of Two Random Variables Is Stochastically Larger Than The Other. Annals of Mathematical Statistics, 18, 50–60.

Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin, 1(6), 80–83.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ If I read you correctly, you are saying that distribution equality is already pre-assumed when I run u_test(x, y) #x~N(0,3) and y~U(-2,2) ? But those textbooks seem to say distribution equality is exactly what the test does, H0: F=G, which made me wonder how does MW differ from Kolmogorov. $\endgroup$ – ABC Analytics Aug 1 at 18:09
  • $\begingroup$ Ross' book seems right to me though. The first two sentences in Mann & Withney's paper state: "Let $x$ and $y$ be two random variables with continuous cumulative distribution functions $f$ and $g$. A statistic $U$ depending on the relative ranks of the $x$'s and $y$'s is proposed for testing the hypothesis $f=g$." If I'm not mistaken, that's exacty what Ross says. $\endgroup$ – COOLSerdash Aug 1 at 18:23
  • $\begingroup$ @ABCAnalytics No, I am not saying that at all. The $U$ test is agnostic to the distribution of each group (other than i.i.d.), spend a minute with the null and alternative hypotheses to land it. You can add further assumptions as I lay out, but that's not what Mann and Whitney wrote. $\endgroup$ – Alexis Aug 1 at 20:12
  • 4
    $\begingroup$ @COOLSerdash it's not the null (at least not expressed in that general form) that determines what is being tested. After all, if you're going to have exchangeability under the null (as we do for a t-test or a test of variances or any number of other tests); any two-sample null could be said to be a test of $f=g$ -- clearly that's not informative. What makes any specific test of interest is the alternatives that it picks up. The Wilcoxon-Mann-Whitney certainly isn't a general test of $f\neq g$, but rather a much more specific one ($P(X<Y) \neq \frac12$ ... $\endgroup$ – Glen_b Aug 2 at 7:27
  • 1
    $\begingroup$ ... (at least in the two-sided case). And that tells us what null we ought to write about the null (its complement). I think Mann and Whitney have done many readers a disservice by merely saying the null is one of exchangeability (once the usual independence is assumed at least), since it tells us nothing of value and misleads people into misunderstanding what the test is for. $\endgroup$ – Glen_b Aug 2 at 7:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.