2
$\begingroup$

I have 5 independent samples which I want to compare, to define if there are significant differences in the distribution of values across samples.

The samples' sizes are: 4562, 1116, 314, 151, 77

I don't assume that the samples are normally distributed, so I've decided to use:

  • Kruskal-Wallis test to have an overall suggestion that some samples are differently distributed
  • If Kruskal-Wallis p is significant ($<0.01$), I run post-hoc Mann-Whitney U test to compare each group to another
  • I am running several tests, so I want to correct for multiple comparisons. I was thinking of using Benjamini-Hochberg FDR

I have several questions:

  • Does the choice of these tests together make sense? Did I miss something?
  • Do I have to test if the tests are sufficiently powered?
  • How can I test for effect size between the samples? Just taking the difference between the medians?
  • If the choice of tests makes sense, since I want to run Mann-Whitney U test on the same data used for Kruskal-Wallis, do I have to account also for that in the correction for multiple comparisons, and if yes how can I do it?
$\endgroup$
2
$\begingroup$

I think your approach makes sense, but I would recommend the Dunn test as a post-hoc test. (No citation on the opinion.) This test is implemented in R, and in SPSS. I suspect it is found in other common software packages.

For effect size statistics for Kruskal-Wallis, I think the most common appropriate ones are epsilon-squared and Freeman's theta. It's important to understand that these effect size statistics are related to the probability that an observation in one group is higher than an observation in another group. That is, they don't measure the absolute difference in observations. For that, you might use the difference in medians, or some kind of standardized version of the difference in medians.

$\endgroup$
  • $\begingroup$ Would you happen to have a bibliographical reference for epsilon-squared in the context of the Kruskal-Wallis test other than the one you've given? I'm desperately trying to find an article that would give a derivation of and a bit more context to the formula... $\endgroup$ – dlukes Jan 7 at 12:13
  • 1
    $\begingroup$ I understand completely. Unfortunately, I don't have anything that has much detail about epsilon-squared. It is also mentioned in King, B.M. and P.J. Rosopa. 2010. Some (Almost) Assumption-Free Tests. In Statistical Reasoning in the Behavioral Sciences, 7th ed. Wiley. $\endgroup$ – Sal Mangiafico Jan 7 at 14:22
  • 1
    $\begingroup$ I checked in Grissom and Kim. 2012. Effect Sizes for Research. Routledge. They don't discuss any effect size statistics specific for the Kruskal-Wallis case. But they do discuss the effect sizes for stochastic dominance in the two sample case (e.g. Mann-Whitney). These statistics include Cliff's delta and Vargha and Delaney's A (which are linearly related). These could be used to address the effect size in an intuitive way. For example, the maximum A between any two groups. Or just pairwise effect sizes. The more I think about this, the more like this approach. $\endgroup$ – Sal Mangiafico Jan 8 at 3:04
  • $\begingroup$ One thing I like about VDA is that if it is immediately understandable. It is the probability of an observation from one group being larger than an observation from the other group. Compare this to to an epsilon-squared of 0.2, which is... "medium"... and means.... ? The only difficulty with VDA is that no effect = 0.5 and those close to 1 or 0 are large. This might be confusing for some readers. Cliff's delta conveys the same information, but setting 0 to no effect and 1 to maximum effect. $\endgroup$ – Sal Mangiafico Jan 8 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.