5
$\begingroup$

Firstly, this is my first post to Cross Validated, so apologies in advance if I have infringed any conventions.

I am writing up a PhD thesis on aspects of biomedical research, and use R for my statistics. However, I always check some of the data against another package in case I have made a coding error.

I have been running a Kruskall-Wallis test with three groups (sample sizes of 5, 8 and 9) and using Dunn's test for the post-hoc pairwise comparisons. The p-values I get for the pairwise comparisons (unadjusted for multiple comparisons) are exactly half what I get in SPSS and GraphPad for the same data. The Kruskal-Wallis p-value reported by the dunn.test package is exactly what I get running the Kruskal-Wallis test itself in R, or in SPSS and Graphpad. There is a warning after I run my code that the dunn.test package was written on a slightly later version of R, but that seems unlikely to be the cause of a factor of 2 variation. Has anyone else come across a similar discrepancy?

$\endgroup$
  • 6
    $\begingroup$ +1 for double checking with a different package. That said, I'm not familiar with Dunn's test, but the documentation for the dunn.test package notes that it uses a one-sided test, which would give you a p value half as large as a corresponding two-sided test (assuming a symmetrical distribution, which would be the only type that would make sense here). Is it possible that the other packages use a two-sided test? $\endgroup$ – S. Kolassa - Reinstate Monica Jul 9 '15 at 11:33
  • $\begingroup$ Good point, I'll look into it. $\endgroup$ – Mark Birtwistle Jul 9 '15 at 12:06
  • $\begingroup$ I got my doubles and my halfs muddled up, so I have amended the question. The p-values in R are smaller than those in the other two packages. $\endgroup$ – Mark Birtwistle Jul 9 '15 at 16:45
  • $\begingroup$ I suspected as much in my comment above. Could you find out whether SPSS & GraphPad do two-sided tests? $\endgroup$ – S. Kolassa - Reinstate Monica Jul 9 '15 at 16:47
  • 1
    $\begingroup$ I have now found a function dunnTest in a rather obscure package called {FSA} (Fisheries Stock Assessment) which runs a two-sided Dunn's test. I think we therefore have enough information to formally answer my question. Do you want to do that @Stephan_Kolassa, or shall I? $\endgroup$ – Mark Birtwistle Jul 10 '15 at 9:48
2
$\begingroup$

The dunn.test package in R uses a one-sided test, whereas SPSS and GraphPad use two-sided tests. There is no facility in the dunn.test package or its function dunn.test() to change to a two-sided test, but the p-values can be multiplied by 2 if a two-sided test is required.

A two-sided Dunn's test is available from the dunnTest() function in package FSA (Fisheries Stock Assessment). This package is not available from CRAN, but can be downloaded by running the code source("http://www.rforge.net/FSA/InstallFSA.R"). It requires an R version more recent than 3.0.2, and I had trouble installing it until I updated the Rcpp package from CRAN. More information on FSA can be found on https://fishr.wordpress.com/fsa/, and documentation on the dunnTest() function can be found on http://www.rforge.net/doc/packages/FSA/dunnTest.html.

Thanks to Stephan Kolassa for his help in resolving this problem.

$\endgroup$
  • $\begingroup$ FSA is a useful package for general users for summarizing data and conducting a few specific tests. It is now available on CRAN. $\endgroup$ – Sal Mangiafico Jul 26 '17 at 15:15
  • $\begingroup$ @SalMangiafico (and Mark Birtwhistle), dunntest does not report one-sided p-values. A one-sided p-value either takes the form $p=P\left(Z \le z\right)$ or the form $p=P\left(Z \ge z\right)$. However, dunntest gives p-values of the form $p=P\left(Z \le |z|\right)$, which is a two-sided test statistic. The correct rejection for this expression of the p-value is to reject Ho if $p\le \alpha/2$. Notice that "Reject Ho if $p\le \alpha/2$" always gives the same test rejection results as "Reject Ho if $2p\le \alpha$". So there is no "irresponsibility" here. $\endgroup$ – Alexis Oct 26 '17 at 23:18
  • $\begingroup$ @SalMangiafico However, enough folks—including yourself—have expressed a desire for dunntest (and dunn.tst in Stata) to express p-values as you expect them, so I am adding an option to do just that (and to make the rejection criteria explicit in the output). Squeaky wheels get the grease. :) $\endgroup$ – Alexis Oct 26 '17 at 23:20
  • 1
    $\begingroup$ As a follow-up on the changes to the dunn.test function, the altp = TRUE option will return what we think of as the two-sided p-value corresponding to the Reject Ho if p <= alpha decision rule. $\endgroup$ – Sal Mangiafico Jul 14 '18 at 14:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.