Multiple comparisons after Kruskal Wallis using the FDR approach. How to compute P values (Dunn or Mann-Whitney)?

I want to allow users of GraphPad Prism to use the False Discovery Rate (FDR) approach to multiple comparisons after nonparametric one-way ANOVA (Kruskal-Wallis). The idea is first to compute an exact (not corrected for multiple comparisons) P value for each comparison, and then to use a standard FDR algorithm to decide which of those P values are small enough for the associated comparison to be designated a "discovery".

How to compute the P value for each comparison?

• Use the Mann-Whitney test.
• Use the Dunn's test, usually used for multiple comparisons, adapted to give an uncorrected P value for each comparison.

Dunn's method uses the rankings from all the groups (and the sample size from all the groups) even when just comparing two groups. Does this add power over doing the Mann-Whitney test for each pair?

In another CV case, Alexis posted "Dunn's test preserves a pooled variance for the tests implied by the Kruskal-Wallis null hypothesis.". But I don't understand how pooled variance is a concern with nonparametric tests.

• What do you mean saying exact (not corrected for multiple comparisons). Was that you computed exact (permutation test) p-values? Or are you about usual asymptotic method p-values? Jul 24, 2015 at 19:02
• It is not correct to do just MW test (with correction) because, as you let drop it, the test will be base on ranks from only those two groups currently compared. The correct approach is to use ranks obtained in the omnibus, Kruskal-Wallis test. Jul 24, 2015 at 19:05
• This link (how SPSS does it) propably answers your question. And it mentions Dunn's method. Jul 24, 2015 at 19:09
• Harvey, by definition, post-hoc multiple comparison tests are tied with the omnibus test performed before them. If the latter is significant then the researcher may ask which groups differ significantly (and thus are the reason why the omnibus was significant). To perform such pairwise tests the researcher not only needs the correction for multiple testing, he as well has to guarantee that the tests' hypotheses are nested within that omnibus test's one, and that they imply each other. Therefore ranks in them should be those ranks that appeared on the omnibus test. Jul 27, 2015 at 16:10
• I want to create a stack of P values, one per comparison, and then use one of the False Discovery Rate methods to decide which are small enough to be tagged as a "discovery". I don't see why the omnibus (Kruskal Wallis) test result is relevant, or why it matters that the individual tests are "nested" in the overall test. I don't see what's wrong with creating the list of P values by doing a bunch of individual Mann-Whitney tests. But I am not sure so am looking for a compelling explanation or proof. Jul 28, 2015 at 0:33

From what I understood of the OP question:

1) He ran a omnibus Kruskal-Wallis with significant results

2) He want to run a pairwise test on all groups and he is in doubt whether to use Mann-Whitney or Dunn's test

3) He want to run his own multiple comparison adjustment procedure, so he needs the uncorrected p-values of each pairwise comparions.

The source of confusion is that Dunn test implemented in GraphPad seems to already include a multiple comparison adjustment (which looks like a Bonferroni adjustment - see http://www.graphpad.com/guides/prism/6/statistics/index.htm?stat_nonparametric_multiple_compari.htm).