What do you do after testing if two non-normal distributions differ?

Question

This is a follow up question to "How do I test if two non-normal distributions differ?

I have 13 distributions, . According to the previous question, I should do the following.

1. Kruskal-Wallis test with 13 samples.
2. Mann-Whitney U pairwise.

I found that all p-values are practically 0. At first, this did not seem right. For example, distributions A and B look very similar. (Subquestion: Should I use alternative = "greater" in R for example? I would like to roughly argue that D is greater than A.) Now, based on another question, I believe that this is due to the fact that my sample sizes are so large. Based on that same question, it appears that I need to calculate the effect size.

I followed these instructions. I obtained tiny effect sizes, for example, 0.00005. How can I interpret these values? I obtain "tiny" effect sizes even when comparing A and D, which to me, look like very different distributions.

To address @whuber's comment, I would like to state that D > L > K > J > ... > [A,B,C]. In other words, that those in D are generally larger than those in L. In the case of A,B,C, I would have to say that there is no significant difference. But again, I would like to an argument based on statistics not just visual observation.

Answer 1

It's not clear to me what that particular 'effect size' is measuring. Its not a measure of the location difference, nor the probability that one sample is greater than the other, nor is it even a form of Z-score. I'm baffled as to why it would be useful; someone may be able to enlighten both of us.

There are two obvious measures of how different the two distributions differ that are related to the Wilcoxon-Mann-Whitney.

1. The first is the Hodges-Lehmann estimate of the median pairwise difference. This is especially suitable if you're interested in a shift alternative*. It's also possible to get a confidence interval around the sample median pairwise difference for the population equivalent (median of the distribution of the difference in the two random variables).

   (R will give you this estimate and a confidence interval when you do wilcox.test if you specify conf.int=TRUE)

   Of course, a shift alternative makes little sense with a bounded variable, so the actual value of the location-difference may be less clear in your case (though the test will still work just fine).

2. The second obvious measure is the proportion of times a value from one sample is less than a value from the other, as an estimate of $P(X<Y)$ (the WMW can be seen as a test of whether $P(X<Y)\neq\frac{1}{2}$).

   This doesn't seem to come with an obvious confidence interval (and I haven't been able to figure one out - indeed it doesn't seem to be distribution-free under the alternative), but it's probably a more directly relevant measure of the difference in the two distributions in your case.

   For it, you just divide the count of times $X<Y$ (i.e. $U_{XY}$, the WMW statistic you already have) by the product of the two sample sizes, $n_1.n_2$, to get the sample proportion $\hat p_{X<Y}$ as an estimate of the corresponding population probability.

See also this answer for more discussion of the two alternative measures of difference.