This is a follow up question to "How do I test if two non-normal distributions differ?

I have 13 distributions, X vs Y. According to the previous question, I should do the following.

  1. Kruskal-Wallis test with 13 samples.
  2. Mann-Whitney U pairwise.

I found that all p-values are practically 0. At first, this did not seem right. For example, distributions A and B look very similar. (Subquestion: Should I use alternative = "greater" in R for example? I would like to roughly argue that D is greater than A.) Now, based on another question, I believe that this is due to the fact that my sample sizes are so large. Based on that same question, it appears that I need to calculate the effect size.

I followed these instructions. I obtained tiny effect sizes, for example, 0.00005. How can I interpret these values? I obtain "tiny" effect sizes even when comparing A and D, which to me, look like very different distributions.

To address @whuber's comment, I would like to state that D > L > K > J > ... > [A,B,C]. In other words, that those in D are generally larger than those in L. In the case of A,B,C, I would have to say that there is no significant difference. But again, I would like to an argument based on statistics not just visual observation.

  • $\begingroup$ What you do depends on the objectives of your data analysis. Could you please describe those in your question? $\endgroup$ – whuber Feb 28 '14 at 21:57
  • $\begingroup$ If I understand your graph correctly, your raw data are usually -1 or 0 or 1 and occasionally 0.5 or -0.5. Also, with your data the median will usually be either -1 or 0. In this situation I'd expect some results to be side-effects of that coarseness of measurement. Modulo the +/- 0.5 values (which in turn may be just averages of other values) your data look almost categorical to me. $\endgroup$ – Nick Cox Mar 4 '14 at 12:43
  • $\begingroup$ @NickCox, you are right, the data is usually 0 or -1. In my data, sets can contain discrete values, [-2,2]. The Y-axis is the average of the sets so it can be any real number within that range. Most of the time the sets contain 1 value so the average is discrete. Do you have any advised for analyzing this type of data. $\endgroup$ – Martin Velez Mar 8 '14 at 5:23

It's not clear to me what that particular 'effect size' is measuring. Its not a measure of the location difference, nor the probability that one sample is greater than the other, nor is it even a form of Z-score. I'm baffled as to why it would be useful; someone may be able to enlighten both of us.

There are two obvious measures of how different the two distributions differ that are related to the Wilcoxon-Mann-Whitney.

  1. The first is the Hodges-Lehmann estimate of the median pairwise difference. This is especially suitable if you're interested in a shift alternative*. It's also possible to get a confidence interval around the sample median pairwise difference for the population equivalent (median of the distribution of the difference in the two random variables).

    (R will give you this estimate and a confidence interval when you do wilcox.test if you specify conf.int=TRUE)

    Of course, a shift alternative makes little sense with a bounded variable, so the actual value of the location-difference may be less clear in your case (though the test will still work just fine).

  2. The second obvious measure is the proportion of times a value from one sample is less than a value from the other, as an estimate of $P(X<Y)$ (the WMW can be seen as a test of whether $P(X<Y)\neq\frac{1}{2}$).

    This doesn't seem to come with an obvious confidence interval (and I haven't been able to figure one out - indeed it doesn't seem to be distribution-free under the alternative), but it's probably a more directly relevant measure of the difference in the two distributions in your case.

    For it, you just divide the count of times $X<Y$ (i.e. $U_{XY}$, the WMW statistic you already have) by the product of the two sample sizes, $n_1.n_2$, to get the sample proportion $\hat p_{X<Y}$ as an estimate of the corresponding population probability.

See also this answer for more discussion of the two alternative measures of difference.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.