4
$\begingroup$

I'm not a statistician, so pardon me for being naive on this subject.

I'm trying to understand if there's any statistically significant difference in the medians of 2 groups. Here are some of the salient features of my groups:

  1. Each of the groups has Millions of observations
  2. Each of the groups is not normally distributed
  3. The observations are continuous
  4. One of the groups has almost 15x the observations in the other group
  5. The groups are mostly independent of each other

If the groups were normally distributed, I could have used the T-test to figure this out.

So this leads me to believe that a Mann-Whitney test would be more useful in this case. But because I have Millions of observations in both the groups, I'm not sure if the Mann-Whitney test results will hold true. In one of the Stack Overflow posts, I read that Mann-Whitney test does not work well with so many observations.

Should I just take much smaller random samples from my 2 groups and perform the Mann-Whitney test many times and then look at the results?

Or is there a better approach to doing this? Any help would be much appreciated.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ In what way does the t-test not work with many observations? What's this SO post that said MW fails with a large sample size? $\endgroup$
    – Dave
    Commented Jun 30, 2020 at 19:46
  • $\begingroup$ @Dave Here's the SO post: stats.stackexchange.com/questions/77359/…. $\endgroup$
    – Patthebug
    Commented Jun 30, 2020 at 19:49
  • $\begingroup$ @Dave Sorry, I was wrong in saying T-test wouldn't work with large sample sizes. I removed it from the post. But my data isn't normally distributed anyway. So a T-test wouldn't work for me. $\endgroup$
    – Patthebug
    Commented Jun 30, 2020 at 19:54
  • 1
    $\begingroup$ If you have millions of observations and you did not do something like randomly allocate the observations to the groups then there will almost certainly be a significantly significant difference between the two medians. But it may be small. So the first thing would be to see what the difference in the sample medians actually is. The second is to think whether that difference is big enough to care even before considering uncertainty. The third might involve developing a confidence interval for that difference. $\endgroup$
    – Henry
    Commented Jun 30, 2020 at 20:08
  • $\begingroup$ @Henry Say the medians are 12 and 15, to me they are significantly different. And yes, both, T-test and MW-test, tell me that the medians are different. I'm not aware of developing a confidence interval approach. I'll have to do more research into it. $\endgroup$
    – Patthebug
    Commented Jun 30, 2020 at 20:32

1 Answer 1

Reset to default
7
$\begingroup$

First, the MW test is not a test of medians (there are some additional assumptions that make it a test of medians, but those same assumptions would also make it a test of means).

The Central Limit Theorem tells us that we can use normal base inference (e.g. the t-test) when the population is not normal, but the sample size is large enough. A sample size of millions is probably big enough that the t-test will give a good enough approximation.

If you still want to compare medians, or still don't trust the t-test then you could use a permutation test (the MW test is a special case of a permutation test) or create a bootstrap confidence interval, or just replace the data with their ranks and do a t-test on the ranks (the MW approximation).

The comment above about looking at the meaningful differences is probably more important than the p-value in this case (with millions of data points you can find extremely small differences to be statistically significant). Confidence intervals on the differences will probably be much more meaningful than p-values at those sizes.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ (+1) even though I think there is even no point in calculating a confidence interval with mio of observations. With such sample sizes, bias dominates the quality of the estimate, while sampling uncertainty is negligible. The c.i. deals only with the latter. $\endgroup$
    – Michael M
    Commented Jun 30, 2020 at 20:41
  • $\begingroup$ So I tried creating confidence intervals. But the intervals are too narrow at 95%. The t-test gives a p-value of 0 consistently. The MW test also gives a p-value of 0 consistently. There's clearly a pattern here. Would it be okay to conclude that statistical tests tend to showcase even small differences as statistically significant, and that we can just rely on medians for such large datasets? $\endgroup$
    – Patthebug
    Commented Jul 1, 2020 at 22:29
  • $\begingroup$ @Patthebug, Yes, with very large sample sizes small differences will still be statistically significant and confidence intervals will be very narrow. That is why we told you to focus on if the differences were meaningful. You should also look for sources of error other than sampling (which is what the CI and p-values account for). $\endgroup$
    – Greg Snow
    Commented Jul 2, 2020 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.