I have 5 independent samples which I want to compare, to define if there are significant differences in the distribution of values across samples.
The samples' sizes are: 4562, 1116, 314, 151, 77
I don't assume that the samples are normally distributed, so I've decided to use:
- Kruskal-Wallis test to have an overall suggestion that some samples are differently distributed
- If Kruskal-Wallis p is significant ($<0.01$), I run post-hoc Mann-Whitney U test to compare each group to another
- I am running several tests, so I want to correct for multiple comparisons. I was thinking of using Benjamini-Hochberg FDR
I have several questions:
- Does the choice of these tests together make sense? Did I miss something?
- Do I have to test if the tests are sufficiently powered?
- How can I test for effect size between the samples? Just taking the difference between the medians?
- If the choice of tests makes sense, since I want to run Mann-Whitney U test on the same data used for Kruskal-Wallis, do I have to account also for that in the correction for multiple comparisons, and if yes how can I do it?