5
$\begingroup$

I want to allow users of GraphPad Prism to use the False Discovery Rate (FDR) approach to multiple comparisons after nonparametric one-way ANOVA (Kruskal-Wallis). The idea is first to compute an exact (not corrected for multiple comparisons) P value for each comparison, and then to use a standard FDR algorithm to decide which of those P values are small enough for the associated comparison to be designated a "discovery".

How to compute the P value for each comparison?

  • Use the Mann-Whitney test.
  • Use the Dunn's test, usually used for multiple comparisons, adapted to give an uncorrected P value for each comparison.

Dunn's method uses the rankings from all the groups (and the sample size from all the groups) even when just comparing two groups. Does this add power over doing the Mann-Whitney test for each pair?

In another CV case, Alexis posted "Dunn's test preserves a pooled variance for the tests implied by the Kruskal-Wallis null hypothesis.". But I don't understand how pooled variance is a concern with nonparametric tests.

|cite|improve this question
$\endgroup$
  • 2
    $\begingroup$ What do you mean saying exact (not corrected for multiple comparisons). Was that you computed exact (permutation test) p-values? Or are you about usual asymptotic method p-values? $\endgroup$ – ttnphns Jul 24 '15 at 19:02
  • 2
    $\begingroup$ It is not correct to do just MW test (with correction) because, as you let drop it, the test will be base on ranks from only those two groups currently compared. The correct approach is to use ranks obtained in the omnibus, Kruskal-Wallis test. $\endgroup$ – ttnphns Jul 24 '15 at 19:05
  • 2
    $\begingroup$ This link (how SPSS does it) propably answers your question. And it mentions Dunn's method. $\endgroup$ – ttnphns Jul 24 '15 at 19:09
  • 2
    $\begingroup$ Harvey, by definition, post-hoc multiple comparison tests are tied with the omnibus test performed before them. If the latter is significant then the researcher may ask which groups differ significantly (and thus are the reason why the omnibus was significant). To perform such pairwise tests the researcher not only needs the correction for multiple testing, he as well has to guarantee that the tests' hypotheses are nested within that omnibus test's one, and that they imply each other. Therefore ranks in them should be those ranks that appeared on the omnibus test. $\endgroup$ – ttnphns Jul 27 '15 at 16:10
  • 1
    $\begingroup$ I want to create a stack of P values, one per comparison, and then use one of the False Discovery Rate methods to decide which are small enough to be tagged as a "discovery". I don't see why the omnibus (Kruskal Wallis) test result is relevant, or why it matters that the individual tests are "nested" in the overall test. I don't see what's wrong with creating the list of P values by doing a bunch of individual Mann-Whitney tests. But I am not sure so am looking for a compelling explanation or proof. $\endgroup$ – Harvey Motulsky Jul 28 '15 at 0:33
4
+50
$\begingroup$

From what I understood of the OP question:

1) He ran a omnibus Kruskal-Wallis with significant results

2) He want to run a pairwise test on all groups and he is in doubt whether to use Mann-Whitney or Dunn's test

3) He want to run his own multiple comparison adjustment procedure, so he needs the uncorrected p-values of each pairwise comparions.

The source of confusion is that Dunn test implemented in GraphPad seems to already include a multiple comparison adjustment (which looks like a Bonferroni adjustment - see http://www.graphpad.com/guides/prism/6/statistics/index.htm?stat_nonparametric_multiple_compari.htm).

Answering:

2) You should use Dunn test. Both the CV answer by @Alexis for Post-hoc tests after Kruskal-Wallis: Dunn's test or Bonferroni corrected Mann-Whitney tests? and this site from XLSAT http://www.xlstat.com/en/products-solutions/feature/kruskal-wallis-test.html agree that Dunn (or Conover-Iman or Steel-Dwass-Critchlow-Fligner ) are the appropriate post-hoc tests after a KW (disclosure - I did not know that until today - have been using Mann-Whitney as post-hoc to KW until today).

3) I did not understand the GaphPad page, but let me point you to the dunn.test package in R does what the OP want. In particular it distinguishes the Dunn test and multiple comparison adjustments, and one can set the adjustment method to "none", which will return the unadjusted p-values. Also notice that among the adjustment procedures there are the Benjamini-Hochberg (95) and the Benjamini-Yekutieli (2001) adjustments that are FDR (maybe one of them is the one the OP is thinking in using).

Let me stress of many of the commentators have been saying - there is no good reason to use the unadjusted p-values EXCEPT to implement your own adjustment procedure - no decision should be made based on the unadjusted p-values.

|cite|improve this answer
$\endgroup$
  • 2
    $\begingroup$ Thanks. I know how to get uncorrected P values from the Dunn's test. That isn't a problem. The question is whether to use those, or use individual P values from Mann-Whitney tests. Your XLSTAT reference also mentions the Dwass and Steel method which in fact begins with individual Mann-Whitney tests (they refer to it as Wilcoxon, but same test). So the question remains: What are the pros and cons of unadjusted Dunn P values vs. individual Mann-Whitney P values (when used as input to a false discovery method of dealing with multiple comparisons)?? $\endgroup$ – Harvey Motulsky Jul 31 '15 at 16:56
  • 2
    $\begingroup$ Thanks for pointing out that the R package does the Dunn test and then offers to use one of two FDR correction methods. I hadn't seen that and it gives me some confidence that we should do the same (actually we have done it, but I was having doubts). But I'd still like a clear explanation of what's wrong with using P values from individual MW tests... $\endgroup$ – Harvey Motulsky Jul 31 '15 at 17:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.