3
$\begingroup$

I am doing a Mann-Whitney U test for two samples with the following descriptives:

Sample 1:
n = 4026
Mean = 15.2
Median = 10.0
Minumum = 1.0
Maximum = 52.0
Standard Deviation = 15.14

Sample 2:
n = 14510
Mean = 15.01
Median = 10.0
Minumum = 1.0
Maximum = 55.0
Standard Deviation = 15.26

My hypothesis is that number of videos watched by one group is greater than the number of videos watched by the second group (Sample 2 > Sample 1)

I get the following result of Mann-Whitney U

(28665796.5, 0.035337297291866368)

The p-value (0.03 < 0.05), which is statistically significant. But the samples have almost identical descriptives. How do I interpret this result?

$\endgroup$
7
$\begingroup$

You run into a common property of the Mann Whitney U test, particularly when larger sample sizes are used or paired tests are performed - the median looks different, but you get a highly significant p-value. This is because your populations maybe have the same median, but have a different distribution.

The following comes from Conroy, The Stata Journal (2012).

The Mann–Whitney test is a test for equality of medians only under the very strong assumption that both of the two distributions are symmetrical about their respective medians or, in the case of asymmetric distributions, that the distributions are of the same shape but differ in location. Thus the common belief that the test compares medians is true only under some implausible circumstances.

Conroy argues that in the case that the researcher wants to compare the median of a distribution, a better test would be the (oldfashioned) median test or quantile regression.

$\endgroup$
  • $\begingroup$ If you would plot a kernel smoothed density estimate of both of them, or compare violin plots, you can see if the empirical distributions are visually distinct. $\endgroup$ – Avraham Jan 4 '17 at 15:53
  • $\begingroup$ As these are counted data, kernel smoothing might confuse as much as reveal. But strong support for idea of showing us entire distributions graphically. $\endgroup$ – Nick Cox Jan 4 '17 at 17:53
2
$\begingroup$

We are told that the hypothesis is Sample 2 $>$ Sample 1. It is actually not clear which hypothesis this is. Is this $H_0$ or $H_1$? I can take a guess and presume that $H_0$ was actually Sample 2 $=$ Sample 1, in which case the $p= 0.035337297291866368$ is the two-tailed probability. If so, and if the probability of Sample 1 $>$ Sample 2 is half that or $p=0.017668648645933184$, then we would reject the $H_0$ hypothesis that Sample 1 $>$ Sample 2 and conclude that Sample 1 $\leq$ Sample 2, which is significant, but not highly significant, so that our confidence in this result is "high enough" but not extremely convincing.

However, there is not enough information to conclude that, and more information as to what $H_0$ and $H_1$ actually were is absolutely required for proper interpretation. For example, it might be that Sample 2 $\leq$ Sample 1 is the result. More information is required for test interpretation!

What we need from OP is what was tested in R

For example, the whole output:

> wilcox.test(mpg ~ am, data=mtcars) 

    Wilcoxon rank sum test with continuity correction 

data:  mpg by am 
W = 42, p-value = 0.001871 
alternative hypothesis: true location shift is not equal to 0 

Warning message: 
In wilcox.test.default(x = c(21.4, 18.7, 18.1, 14.3, 24.4, 22.8,  : 
cannot compute exact p-value with ties 

Now the statistic for which ever the result is, is the Mann-Whitney U-stat, see link. This is not a median or a mean difference test. In general terms, the Mann-Whitney U test is a test of location, where by location we mean something more general than mean or median.

$\endgroup$
  • $\begingroup$ Some extravagance there in # of decimal places! $\endgroup$ – Nick Cox Jan 4 '17 at 17:56
  • $\begingroup$ @NickCox That is what was given, truncation would obscure that. Sure, only a few are significant, but wanting to round or truncate in base 10 is arbitrary as well. Perhaps $p=0.018\bar{0}$ make you feel "good" in a human way, it is however, less correct in absolute terms. $\endgroup$ – Carl Jan 4 '17 at 18:20
  • $\begingroup$ For problems like this 3 dp seem adequate to me. With this pair of sample sizes, the calculation is presumably not exact even in principle, noot that anyone should obviously need to care about that. You know all this; it's setting an example of good practice to all that is crucial! $\endgroup$ – Nick Cox Jan 4 '17 at 18:42
  • $\begingroup$ @NickCox Yes, but even 3 decimal places is probably excessive. There are several points I was making, 1) exactly what I was doing using the numbers as a fingerprint, not as a recommendation 2) One does not have enough information to even determine which direction the probability is in, never mind how many decimal places are significant. 3) Without more information, I did not want to give a 'human' or 'quotable' result, we just are not there at the moment. $\endgroup$ – Carl Jan 4 '17 at 19:00
1
$\begingroup$

Probably your samples are large enough to have a difference in the upper or lower ranges of the descriptive. If you could provide some additional information on the percentiles you might see differences there.

So what I would suggest is that you look at the 10th, 20th, 30th, 40th, 50th, 60th, 70th, 80, and 90th percentiles within the two groups (next to the minimum=0th, and maximum=100th percentile). What you could see is that even though the ranges within the two groups are the same, one group might have a larger proportion of participants with a lower or respectively higher descriptive value. As the Mann-Whitney U performs its significance testing on ranked data instead of the actual values, a significant difference is not always seen in the median.

Another tip: it is always nice to see a median with the accompanying 25th and 75th percentiles as a measure of the data's distribution.

$\endgroup$
  • $\begingroup$ Thanks for the comment. I am fairly new to statistical analysis. My question is, since I have fairly large N's for both my samples, can I just use a t-test? How is that different from using Mann-Whitney U? $\endgroup$ – SZA Jan 5 '17 at 17:05
  • $\begingroup$ Large N's can seem an 'easy out' reason to use a T-test, and ignore the non-normality in your data. and yes, this might be an option in skewed data as well. However, a large N does not necessarily mean you comply to the assumptions of a 'simple' T-test. These are: homogeneity of variance (similar variance in both groups) and normality of the error distribution. Note that using a non-parametric test such as the Mann-Whitney U only (partly) removes the second assumption and does not change the first! Furthermore, it has assumptions of its own as well. $\endgroup$ – IWS Jan 6 '17 at 9:35
1
$\begingroup$

This brings to mind Anscombe's famous quartet where the mean of $x$ and $y$, the sample variances of $x$ and $y$, the correlation between $x$ and $y$, and the linear regression and intercept are the same for all four data sets to at least 2 (and sometimes 3) decimals, yet graphing them is eye opening:

enter image description here

As the other answers said, it really pays to visually inspect the data, using a kernel-smoothed density, violin plots, or even box-and-whiskers to see the actual data behavior instead of just trying to identify a data set by its summary statistics. In your case, as said, you probably have similar medians and endpoints, but a different distribution of the data points. Graph them and see!

$\endgroup$
  • 1
    $\begingroup$ I can't see how this is related to the question..? $\endgroup$ – Tim Jan 4 '17 at 16:04
  • $\begingroup$ @Tim Because the OP asked how is it possible that two data sets with similar summary statistics can test as different under Mann-Whitney, and the answer is summary statistics can be misleading as they hide distributional information. The classic example of this is Anscombes Quartet. $\endgroup$ – Avraham Jan 4 '17 at 16:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.