Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I conducted an experiment, testing three conditions with 27 participants in a repeated measures design. Since the data is not normally distributed I used a Friedman test that reported a p-value of 0.1, a slight disappointment given how much time I spent in the lab. Being disappointed but curious, I still performed Wilcoxon's signed rank test as post-hoc analyses for the three pairs. The interesting thing is that these tests reported p-values 0.002, 0.004 and 0.25, respectively. Even after a - conservative - Bonferroni correction two of the p's are still significant.

Now, besides the ethical/scientific issues surrounding digging around in the data like that: why are the p-values of the Friedman and the Wilcoxon tests so far apart? And: how do I interpret these findings? They seem to contradict each other.

share|improve this question
2  
You have repeated measures, but neither Friedman nor Wilcoxon pays any attention to the structure implied. The main answer to your post is that different questions get different answers; that is not a contradiction. –  Nick Cox Jun 19 '13 at 10:04
    
The same kind of thing can happen with other post-hoc tests. –  Glen_b Jun 19 '13 at 10:08
1  
Note that the Friedman test in most situations has far less power than Wilcoxon-type tests. A generalization of the Wilcoxon test that is more suited to your situation is the mixed effects proportional odds model. But a quick and only slightly dirty approach is to use multiple Wilcoxon tests as you have done. On a more general note, even if the best available method resulted in a large P-value, the time was not wasted in the lab; it's just that many people are biased against "negative" studies. It is possible to learn a great deal from negative studies. –  Frank Harrell Jun 19 '13 at 11:40
    
1) In Friedman, you rank within all 3 conditions at once, but in pairwise comparisons you rank only within 2 being currently compared. So the results need not to agree. 2) Even more: Friedman test is not the extension of Wilcoxon test over to 3+ conditions. Friedman is - I'd say - closer to be an "extended" sign z-test. –  ttnphns Jun 19 '13 at 11:42
1  
@xmjx be sure not to make the "absence of evidence is not evidence for absence" mistake. You would need to base you "if there is a difference" assessment on confidence intervals excluding meaningful differences, not on large P-values. –  Frank Harrell Jun 19 '13 at 13:54

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.