Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I conducted an experiment, testing three conditions with 27 participants in a repeated measures design. Since the data is not normally distributed I used a Friedman test that reported a p-value of 0.1, a slight disappointment given how much time I spent in the lab. Being disappointed but curious, I still performed Wilcoxon's signed rank test as post-hoc analyses for the three pairs. The interesting thing is that these tests reported p-values 0.002, 0.004 and 0.25, respectively. Even after a - conservative - Bonferroni correction two of the p's are still significant.

Now, besides the ethical/scientific issues surrounding digging around in the data like that: why are the p-values of the Friedman and the Wilcoxon tests so far apart? And: how do I interpret these findings? They seem to contradict each other.

share|improve this question
2  
You have repeated measures, but neither Friedman nor Wilcoxon pays any attention to the structure implied. The main answer to your post is that different questions get different answers; that is not a contradiction. –  Nick Cox Jun 19 '13 at 10:04
    
The same kind of thing can happen with other post-hoc tests. –  Glen_b Jun 19 '13 at 10:08
1  
Note that the Friedman test in most situations has far less power than Wilcoxon-type tests. A generalization of the Wilcoxon test that is more suited to your situation is the mixed effects proportional odds model. But a quick and only slightly dirty approach is to use multiple Wilcoxon tests as you have done. On a more general note, even if the best available method resulted in a large P-value, the time was not wasted in the lab; it's just that many people are biased against "negative" studies. It is possible to learn a great deal from negative studies. –  Frank Harrell Jun 19 '13 at 11:40
    
1) In Friedman, you rank within all 3 conditions at once, but in pairwise comparisons you rank only within 2 being currently compared. So the results need not to agree. 2) Even more: Friedman test is not the extension of Wilcoxon test over to 3+ conditions. Friedman is - I'd say - closer to be an "extended" sign z-test. –  ttnphns Jun 19 '13 at 11:42
1  
@xmjx be sure not to make the "absence of evidence is not evidence for absence" mistake. You would need to base you "if there is a difference" assessment on confidence intervals excluding meaningful differences, not on large P-values. –  Frank Harrell Jun 19 '13 at 13:54

1 Answer 1

As far as i know, you can't run Wilcoxon reliably, if Friedman test showed no significant difference.

share|improve this answer

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

4  
Welcome to the site. At present this is more of a comment than an answer. Would you mind expanding it a little? –  gung Sep 17 at 3:52
2  
At the least you should explain what you intend by 'reliably'. What is impacted? How? –  Glen_b Sep 17 at 5:33
    
This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post - you can always comment on your own posts, and once you have sufficient reputation you will be able to comment on any post. –  kjetil b halvorsen Sep 17 at 13:16

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.