2
$\begingroup$

This question already has an answer here:

I am comparing two categories (A and B) of social media posts with the corresponding number of likes for each one. I am taking samples of equal size from each category at random and performing a Mann-Whitney U-test using scipy.stats module in Python.

I have chosen U-test since taking the mean of A or B data does not make a lot of sense in my case and I have been relying on medians so far for the comparisons.

I was performing the test with sample sizes in range of 20-100 which gave the expected results that the two categories were similar. So I decided to try larger samples. With sample sizes >= 200, the p-value of the U-statistic was < .05, which could indicate that the distributions of the two samples might be different (at alpha = 5%). However, the visual analysis of the samples (n = 200) shows otherwise, i.e. the difference in the two distributions is very minuscule (see below). Is there something I'm not getting/doing wrong/misinterpreting? Thanks a bunch in advance.

Relative freq. histogram

enter image description here

$\endgroup$

marked as duplicate by Glen_b -Reinstate Monica Feb 10 '17 at 23:15

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ You say: "I have chosen U-test since taking the mean of A or B data does not make a lot of sense in my case and I have been relying on medians so far for the comparisons" ,,, but the Mann-Whitney is not a comparison of medians. [Indeed it's possible for the sample medians to differ in one direction and for the U test to reject a one-tailed test in the opposite direction.], -- both this issue and the point of your question (which is not particular to the Mann-Whitney, it happens with any consistent test) are discussed in many answers on site. $\endgroup$ – Glen_b -Reinstate Monica Feb 10 '17 at 23:13
  • $\begingroup$ Beside the indicated duplicate, see for example this one specifically about the Mann-Whitney -- which question was in the "Related Questions" in the sidebar when I started typing this, so it was probably suggested to you when you posted ... though it will now move to "Linked" since I linked it) $\endgroup$ – Glen_b -Reinstate Monica Feb 10 '17 at 23:17
  • $\begingroup$ @Glen_b Hey, what about statistics.laerd.com/premium-sample/mwut/…, i.e. comparing the medians under the assumption that both distributions have the same shape. $\endgroup$ – KJ7 Feb 13 '17 at 12:25
  • $\begingroup$ It's true, but if you add an assumption that strong, it's also a test for means, and lower quartiles, and 90th percentiles, and midranges and trimeans and midhinges and .... almost any other location measure (whenever the relevant population quantities exist). So I still wouldn't specifically call it a test for medians in that case either; in that situation it's a test for whatever location measure you want. Note that with an ordinary two-sample t-test (under its usual assumptions), by the same kind of argument used at your link, it's also a test for medians. $\endgroup$ – Glen_b -Reinstate Monica Feb 13 '17 at 12:36
2
$\begingroup$

Your sample size is quite large, and will certainly detect small changes in the distribution. Unless the two samples are exactly identical (never happen in real-life), given enough number of samples, you statistical test would always give you a significant p-value.

I can understand what you want to do. You want to prove the two samples come from the same population and are identical statistically. However, they are different as you can see in the graph. The distribution in group A and B are close but not identical. There is no reason why Mann-Whitney test wouldn't give you a significant result.

Your difference may be practically insignificant but statistically significant.

References:

$\endgroup$
  • 1
    $\begingroup$ In addition to this good answer: the interpretation of "p>0.05 shows that there is no difference" mentioned by the OP is wrong - perhaps he needs an equivalenve test. $\endgroup$ – Björn Feb 10 '17 at 7:22
  • 1
    $\begingroup$ Might also be worth pointing out that MW does not compare medians but stochastic equality. $\endgroup$ – mdewey Feb 10 '17 at 12:29
  • 1
    $\begingroup$ @mdewey My comments were more about general statistical testing. But you're right. $\endgroup$ – SmallChess Feb 10 '17 at 12:29
  • $\begingroup$ @Björn I said "...the p-value of the U-statistic was < .05, which could indicate that the distributions of the two samples might be different (at alpha = 5%)...". Are you saying that's incorrect? I thought p values below alpha allow you to reject the H0 that the samples come from the same distribution, no? $\endgroup$ – KJ7 Feb 10 '17 at 14:13
  • $\begingroup$ @StudentT Thanks. So, basically I'm getting significant p-values because the test is overpowered by the large sample sizes? Do you think it would make sense, given the large amount of data that I gave, to perform several U-tests on a number of randomly drawn samples (n = 20) to confirm the alternative hypothesis? Another reason why I'm using U-test is because I need to automate the comparisons between the different categories to determine which one is statistically significant. $\endgroup$ – KJ7 Feb 10 '17 at 14:18

Not the answer you're looking for? Browse other questions tagged or ask your own question.