6
$\begingroup$

I'm wanting to compare scores derived from a reaction time task between two groups with unequal sizes (G1 = 78; G2 = 23). However, when I run the U test it tells me there is no significant difference, U = 897.00, z = .000, P = 1.00. How am I getting significance of 1.00 when there is at least some difference between the means (.21 vs .20)? I was expecting no difference, but these stats are different from what I got from t-testing. I am running the U-test to account for unequal group sizes and non-normal data on several different variables, but is a z-statistic of .000 and P value of 1.00 accurate to report?

Thanks in advance.

$\endgroup$

2 Answers 2

7
$\begingroup$

How can a Mann-Whitney U-Test return a p = 1.00 for unequal means?

Because it's not a test for means.

I'm wanting to compare scores derived from a reaction time task between two groups with unequal sizes (G1 = 78; G2 = 23). However, when I run the U test it tells me there is no significant difference, U = 897.00, z = .000, P = 1.00. How am I getting significance of 1.00 when there is at least some difference between the means (.21 vs .20)?

Because it doesn't compare means! (Nor does it compare medians, in spite of many books saying otherwise)

Even though the difference in means is slightly different from 0, the thing that the Mann-Whitney looks at* turned out to be 0.

* whether conceived in terms of average rank or as two-sample Hodges-Lehmann difference.

(See this answer, for example)

I was expecting no difference, but these stats are different from what I got from t-testing.

If they were always the same, you wouldn't need two different tests.

I am running the U-test to account for unequal group sizes and non-normal data on several different variables, but is a z-statistic of .000 and P value of 1.00 accurate to report?

I'm not sure what you mean by "accurate" but I'd just report those figures; they're certainly possible.

$\endgroup$
4
  • $\begingroup$ You say what the Mann-Whitney U-Test doesn't test for, but you don't really clarify what it does test for. This answer would be improved by talking about what, exactly, "the thing that the Mann-Whitney looks at" is, why people might be mislead to thinking it's testing for means/medians, and in what situation it is useful, if it's not to test for differences in means/medians. $\endgroup$
    – R.M.
    Dec 1, 2016 at 2:42
  • 1
    $\begingroup$ @R.M I do say what it tests for in this sentence "whether conceived in terms of average rank or as two-sample Hodges-Lehmann difference". More detail on what it does test for can be found in other answers already on site, but since that wasn't the main question here going into it in detail didn't seem to be critical to an answer. You seem to be asking me to go far beyond what the question asks to answering different questions but if this question had instead actually asked those things, it would close as a duplicate. However, when I can I will locate a link or two with additional details. $\endgroup$
    – Glen_b
    Dec 1, 2016 at 2:47
  • $\begingroup$ @Glen_b Added some text to answer that may address this. $\endgroup$
    – Carl
    Dec 1, 2016 at 3:29
  • 1
    $\begingroup$ @Glen_b Links to other posts would be fine. (One of the reasons I wrote my comment was that there was zero links to additional reading/support.) The question is "How can a Mann-Whitney U-Test return a p = 1.00 for unequal means?" A good answer to that question would address the misconception which prompted it - especially if, as you point out, it (or similar misconceptions) are in "many books". It's a little unfair to just throw readers to the wild when you freely admit there's a bunch of wrong information out there. $\endgroup$
    – R.M.
    Dec 1, 2016 at 13:16
1
$\begingroup$

The Mann-Whitney U test is a non-parametric test meaning that it is, loosely put, counting up hits and misses for rankings with the point being that the number of outcomes is countable as opposed to real continuous like a $t$-statistic. With some simplification the number of outcomes is $\frac{n(n+1)}{2}$ or the number of combinations of $n$ objects taken two at a time, and in practice, some of those combinations yield a $p=1$. What this does is to limit the number of possible probability values a U-statistic can output such that a perfect $p=1$ merely means that no difference was detected as the countable events matched. The $p=1$ is an accidental outcome, that is, it does not have the same meaning as it would in a deterministic system, and even suspected to be deterministic systems suffer the black swan problem. That is, just because the probability of seeing only white swans might seem like $p=1$, doesn't make that estimate hold for a larger dataset. That is, one needs a solid physical reason for calling a system deterministic, and guesswork alone cannot provide that. As it is, the outcomes for the Mann-Whitney U test are countable, each outcome is only approximate.

Couldn't help but notice the discussion above. Here is a worked example of what the Mann-Whitney U test does. In that file, one can see that the Mann-Whitney U test tests differences in location of the data sets using a fairly comprehensive comparison of rankings. The concept of location of data is a more general one than mean or median. The best measure of data location can be thought of as the minimum variance unbiased estimator of data location MVUE. For example, for a uniform distribution of unknown population endpoints, the average extreme value $\frac{min(x)+max(x)}{2}$ of $x$-values is a lower variance estimator of location than either the mean or median, despite the fact that for uniform distributions all three measures tend to the same location in the limit as the number of observations increases unbounded large.

The Mann-Whitney U test, has as its most important feature is that it is an unbiased test of difference of location as it ignores non-normality. Finally, the difference of location measure of the Mann-Whitney U test is...wait for it...the U-statistic.

The theory of U-statistics allows a minimum-variance unbiased estimator to be derived from each unbiased estimator of an estimable parameter (alternatively, statistical functional) for large classes of probability distributions. If $f(x_1, x_2) = |x_1 - x_2|$, the U-statistic is the mean pairwise deviation $f_n(x_1,\ldots, x_n) = \sum_{i\neq j} |x_i - x_j| / (n(n-1))$, defined for $n\ge 2$. That theory explains that the U-statistic is MVUE for the median of data triplets and not of $n$ data samples.

$\endgroup$
6
  • $\begingroup$ Wouldn't there be more than one way of getting the least-discrepant test statistic? (the one with a p-value of 1) $\endgroup$
    – Glen_b
    Dec 1, 2016 at 1:59
  • $\begingroup$ @Glen_b Yes, but I simplified, maybe 2. But only one is chosen no? $\endgroup$
    – Carl
    Dec 1, 2016 at 2:03
  • $\begingroup$ Consider a Mann-Whitney with n1=3, n2=4 (and no ties). The middle-most statistic (the one with a p-value of 1) occurs in 5 ways. $\endgroup$
    – Glen_b
    Dec 1, 2016 at 2:10
  • $\begingroup$ @Glen_b Taking you word for it, it's a messy calculation. $\endgroup$
    – Carl
    Dec 1, 2016 at 3:37
  • 2
    $\begingroup$ Not especially. w.l.o.g., taking as test statistic the sum of ranks in the first group, you can write all 35 sets of the possible ranks in gp1: 123,124,125,126,127,134,135,136,137,145,146,147,156,157,167,234,235,236,237,245,246,247,256,257,267,345,346,347,356,357,367,456,457,467,567. The rank sums run from 6 to 18 and the middle sum, 12, occurs 5 times with the rank sets 147,156,237,246,345 . That only took a couple of minutes at most, but if you're feeling lazy, R will do it in no time: table(colSums(combn(7,3))) $\endgroup$
    – Glen_b
    Dec 1, 2016 at 4:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.