Mann-Whitney U for two distributions with equal mean

Question

I am doing a Mann-Whitney U test for two samples with the following descriptives:

Sample 1:
n = 4026
Mean = 15.2
Median = 10.0
Minumum = 1.0
Maximum = 52.0
Standard Deviation = 15.14

Sample 2:
n = 14510
Mean = 15.01
Median = 10.0
Minumum = 1.0
Maximum = 55.0
Standard Deviation = 15.26

My hypothesis is that number of videos watched by one group is greater than the number of videos watched by the second group (Sample 2 > Sample 1)

I get the following result of Mann-Whitney U

(28665796.5, 0.035337297291866368)

The p-value (0.03 < 0.05), which is statistically significant. But the samples have almost identical descriptives. How do I interpret this result?

Answer 1

You run into a common property of the Mann Whitney U test, particularly when larger sample sizes are used or paired tests are performed - the median looks different, but you get a highly significant p-value. This is because your populations maybe have the same median, but have a different distribution.

The following comes from Conroy, The Stata Journal (2012).

The Mann–Whitney test is a test for equality of medians only under the very strong assumption that both of the two distributions are symmetrical about their respective medians or, in the case of asymmetric distributions, that the distributions are of the same shape but differ in location. Thus the common belief that the test compares medians is true only under some implausible circumstances.

Conroy argues that in the case that the researcher wants to compare the median of a distribution, a better test would be the (oldfashioned) median test or quantile regression.

Answer 2

We are told that the hypothesis is Sample 2 $>$ Sample 1. It is actually not clear which hypothesis this is. Is this $H_0$ or $H_1$? I can take a guess and presume that $H_0$ was actually Sample 2 $=$ Sample 1, in which case the $p= 0.035337297291866368$ is the two-tailed probability. If so, and if the probability of Sample 1 $>$ Sample 2 is half that or $p=0.017668648645933184$, then we would reject the $H_0$ hypothesis that Sample 1 $>$ Sample 2 and conclude that Sample 1 $\leq$ Sample 2, which is significant, but not highly significant, so that our confidence in this result is "high enough" but not extremely convincing.

However, there is not enough information to conclude that, and more information as to what $H_0$ and $H_1$ actually were is absolutely required for proper interpretation. For example, it might be that Sample 2 $\leq$ Sample 1 is the result. More information is required for test interpretation!

What we need from OP is what was tested in R

For example, the whole output:

> wilcox.test(mpg ~ am, data=mtcars) 

    Wilcoxon rank sum test with continuity correction 

data:  mpg by am 
W = 42, p-value = 0.001871 
alternative hypothesis: true location shift is not equal to 0 

Warning message: 
In wilcox.test.default(x = c(21.4, 18.7, 18.1, 14.3, 24.4, 22.8,  : 
cannot compute exact p-value with ties 

Now the statistic for which ever the result is, is the Mann-Whitney U-stat, see link. This is not a median or a mean difference test. In general terms, the Mann-Whitney U test is a test of location, where by location we mean something more general than mean or median.

Answer 3

Probably your samples are large enough to have a difference in the upper or lower ranges of the descriptive. If you could provide some additional information on the percentiles you might see differences there.

So what I would suggest is that you look at the 10th, 20th, 30th, 40th, 50th, 60th, 70th, 80, and 90th percentiles within the two groups (next to the minimum=0th, and maximum=100th percentile). What you could see is that even though the ranges within the two groups are the same, one group might have a larger proportion of participants with a lower or respectively higher descriptive value. As the Mann-Whitney U performs its significance testing on ranked data instead of the actual values, a significant difference is not always seen in the median.

Another tip: it is always nice to see a median with the accompanying 25th and 75th percentiles as a measure of the data's distribution.

Answer 4

This brings to mind Anscombe's famous quartet where the mean of $x$ and $y$, the sample variances of $x$ and $y$, the correlation between $x$ and $y$, and the linear regression and intercept are the same for all four data sets to at least 2 (and sometimes 3) decimals, yet graphing them is eye opening:

As the other answers said, it really pays to visually inspect the data, using a kernel-smoothed density, violin plots, or even box-and-whiskers to see the actual data behavior instead of just trying to identify a data set by its summary statistics. In your case, as said, you probably have similar medians and endpoints, but a different distribution of the data points. Graph them and see!