I am using the Dunn test (the package Dunn.test in R) as a post-hoc test to my Kruskal Wallis analysis. My alpha value is 0.05, so I would reject the Ho for p - values <= 0.05.

When printing my results, R prints the note:

alpha = 0.05
Reject Ho if p <= alpha/2

Does anybody happen to know why R advises me that I should reject Ho if the p-value is 0.025 even at a set alpha value of 0.05? (For clarity: why is R using this particular decision rule of alpha/2 instead of just alpha?)

My guess would be that it has something to do with testing one- or two sided. I couldn't find if the default of Dunn.test and therewith the given results is for a one or two sided approach.

  • 1
    $\begingroup$ I believe it's because that's how the test was originally written. The author of the package is trying to be as faithful to the test's author as possible, while trying to alert the user of how the test is designed. $\endgroup$ – Sal Mangiafico Mar 28 '19 at 21:43
  • $\begingroup$ Hi Sarah, welcome to CV. Incidentally, the documentation of dunn.test, which explicitly answers this question, also includes my contact information, and I am responsive to information requests, bug ix requests, feature requests, etc. $\endgroup$ – Alexis Mar 30 '19 at 15:53
  • 2
    $\begingroup$ Oh wow, thank you for your replies. The hint towards the original paper and original threshold helped me to clarify my questions! Sorry for the really stupid/confusing typo, it is of course the alpha of 0.05! ... and a big sorry for my "poor sentence structure" I am not a native speaker but trying by best to improve in the future! $\endgroup$ – Sarah Mar 30 '19 at 18:11

With dunn.test you have an argument altp which sets how the p-value will be expressed. If in function call you set altp=TRUE, then the p values will be expressed in alternative format.

Test default is to express p-value = P(Z ≥ |z|), and reject Ho if p ≤ α/2. So what you describe sounds like normal behaviour. You still can change it - if the altp option is used, p-values are instead expressed as p-value = P(|Z| ≥ |z|), and Ho is rejected if p ≤ α.

As it´s said in documentation, both expressions should anyway give identical test results so the use of altp is therefore merely a semantic choice".

| cite | improve this answer | |
  • 2
    $\begingroup$ I am the author of dunn.test (R) and dunntest (Stata), and I approve of this answer. $\endgroup$ – Alexis Mar 30 '19 at 15:38

If you read the original O. J. Dunn (1964) paper, decisions in the test are made using a threshold based on α / 2. My understanding is that the author of the dunn.test package wrote the dunn.test function to be faithful to the test as it was designed in the original paper.

I find reporting the p values as relative to an α / 2 threshold to be confusing for most users, but as the OP points out, the function output tells you what decision rule to use. And as @Oka points out, the function includes an option to report results relative to an α threshold.

Therefore, if the altp = TRUE option is used, the results will match those from the dunnTest function in the FSA package.



A = rnorm(20, 10, 3)
Y = c(A, A+2, A+4)
G = factor(rep(c("A", "B", "C"), 1, each = length(Y)/3))

plot(Y ~ G)

dunnTest(Y ~ G, method="holm")

   ### Dunn (1964) Kruskal-Wallis multiple comparison
   ### p-values adjusted with the Holm method.
   ###   Comparison         Z     P.unadj       P.adj
   ### 1      A - B -1.602483 0.109048909 0.218097818
   ### 2      A - C -3.204965 0.001350787 0.004052361
   ### 3      B - C -1.602483 0.109048909 0.109048909

dunn.test(Y, G, method="holm", altp=TRUE)

   ### Comparison of Y by G                              
   ### (Holm)
   ### Col Mean-|
   ### Row Mean |          A          B
   ### ---------+----------------------
   ###        B |  -1.602482
   ###          |     0.2181
   ###          |
   ###        C |  -3.204965  -1.602482
   ###          |    0.0041*     0.1090
   ### alpha = 0.05
   ### Reject Ho if p <= alpha
| cite | improve this answer | |
  • 1
    $\begingroup$ The dunnTest function in FSA is based on my code, so no surprises that they match. :) $\endgroup$ – Alexis Mar 30 '19 at 16:03
  • $\begingroup$ @Alexis :) .... $\endgroup$ – Sal Mangiafico Mar 30 '19 at 16:05

The proper decision rule is to reject Ho when the pvalue does not exceed alpha, so when alpha and p-value are equal, you still reject Ho.

This is consistent with what R is telling you.

| cite | improve this answer | |
  • 1
    $\begingroup$ I think you didn't understand what the question is asking. $\endgroup$ – Sal Mangiafico Mar 30 '19 at 14:27
  • $\begingroup$ How so? "Does anybody happen to know why R advises me that I should reject Ho if the p-value is 0.025 even at a set alpha value of 0.5?" Either the "0.5" is a typo and should be .05, or the user is asking (per sentence structure) why R tells her to reject Ho if the p-value is .025 when alpha is .5. The only other option I can interpret (loosely, possible due to poor sentence structure) is that R is telling the OP to reject if p < .025 despite the user specifying alpha .5 and R is incorrectly showing .025 instead of .25. The last option, if correct, is unclear due to sentence structure. $\endgroup$ – LSC Mar 30 '19 at 14:40
  • 1
    $\begingroup$ Yes, the 0.5 is a typo. The question is about why this function suggests that the decision rule to be followed is reject when p <= alpha / 2, not reject when p <= alpha, as is typical. So the OP might better have written Does anybody happen to know why R advises me that I should not reject Ho if the p-value is 0.04 even at a set alpha value of 0.05? . $\endgroup$ – Sal Mangiafico Mar 30 '19 at 14:54
  • $\begingroup$ So, poor wording and typographical error. The one-tail alpha is still alpha just adjusted for the tailedness of the test, so typical rejection rules still apply. It may have been even better for OP to have asked if and why the test documentation uses a one-tailed allocation for alpha in the decision rule. $\endgroup$ – LSC Mar 30 '19 at 21:25
  • 1
    $\begingroup$ Yes, you could phrase it that way. $\endgroup$ – Sal Mangiafico Mar 31 '19 at 13:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.