I conducted an experiment, testing three conditions with 27 participants in a repeated measures design. Since the data is not normally distributed I used a Friedman test that reported a p-value of 0.1, a slight disappointment given how much time I spent in the lab. Being disappointed but curious, I still performed Wilcoxon's signed rank test as post-hoc analyses for the three pairs. The interesting thing is that these tests reported p-values 0.002, 0.004 and 0.25, respectively. Even after a - conservative - Bonferroni correction two of the p's are still significant.

Now, besides the ethical/scientific issues surrounding digging around in the data like that: why are the p-values of the Friedman and the Wilcoxon tests so far apart? And: how do I interpret these findings? They seem to contradict each other.

  • 5
    $\begingroup$ You have repeated measures, but neither Friedman nor Wilcoxon pays any attention to the structure implied. The main answer to your post is that different questions get different answers; that is not a contradiction. $\endgroup$
    – Nick Cox
    Commented Jun 19, 2013 at 10:04
  • $\begingroup$ The same kind of thing can happen with other post-hoc tests. $\endgroup$
    – Glen_b
    Commented Jun 19, 2013 at 10:08
  • 2
    $\begingroup$ Note that the Friedman test in most situations has far less power than Wilcoxon-type tests. A generalization of the Wilcoxon test that is more suited to your situation is the mixed effects proportional odds model. But a quick and only slightly dirty approach is to use multiple Wilcoxon tests as you have done. On a more general note, even if the best available method resulted in a large P-value, the time was not wasted in the lab; it's just that many people are biased against "negative" studies. It is possible to learn a great deal from negative studies. $\endgroup$ Commented Jun 19, 2013 at 11:40
  • 1
    $\begingroup$ 1) In Friedman, you rank within all 3 conditions at once, but in pairwise comparisons you rank only within 2 being currently compared. So the results need not to agree. 2) Even more: Friedman test is not the extension of Wilcoxon test over to 3+ conditions. Friedman is - I'd say - closer to be an "extended" sign z-test. $\endgroup$
    – ttnphns
    Commented Jun 19, 2013 at 11:42
  • 1
    $\begingroup$ @xmjx be sure not to make the "absence of evidence is not evidence for absence" mistake. You would need to base you "if there is a difference" assessment on confidence intervals excluding meaningful differences, not on large P-values. $\endgroup$ Commented Jun 19, 2013 at 13:54


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.