I am doing a study on 3 drugs, comparing response pre-post treatment. My objective is to know if these drugs are effective and which one is better. I used non parametric tests since the results weren't normally distributed and were non transformable. Two drugs were effective with a significant difference on the Wilcoxon signed test and a third showed no significant difference. Yet when comparing the post-results using the Kruskal-Wallis test, no significant difference was observed and the pre-results showed non significant differences between these groups using the Kruskal-Wallis test as well. Why is that? Did I choose the wrong tests?
- Anybody can ask a question
- Anybody can answer
- The best answers are voted up and rise to the top
What you should always keep in mind, is that The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant -- there is a nice paper with this title by Gelman and Stern, which I link to here, but the idea is very simple. Here is how they start explaining it:
In your case, when you conduct three separate Wilcoxon tests, you might get p-values e.g. $0.045$, $0.045$, and $0.055$. First two are "significant" according to the common $p<0.05$ criterion, and the third one is not. However, the difference between p-values is tiny, and so it is very well possible that if you compare three groups between each other, then you will fail to get any significant difference. Which seems to be exactly your case.
In addition: doing Kruskal-Wallis on the pre and post measures separately is probably not the best approach. You can subtract pre from post and do one Kruskal-Wallis on these differences. It is of course still possible (as I explain above) that you will not get a significant difference, but this is a more correct approach.
Just to stress it again: if one drug comes out with significant pre-post difference and another one with insignificant, it is (by itself) no reason whatsoever to believe that one drug is better than another. Unfortunately, it is a very widespread mistake.
Those tests are all testing different hypotheses. I see no reason to be surprised that they don't agree on the answers to different questions. In addition, tests of the same hypotheses, but that work differently or make different assumptions can yield inconsistent results if the p-values are all around the significance threshold alpha.
Regarding what you should do here, if you have pre and post data on three drugs and want to see if they differ, I might subtract pre from post and run a Kruskal-Wallis test on the three sets of differences. That will check to see if the pre-post changes differ amongst the drugs. If that is significant, you could use Mann-Whitney U-tests with different pairs of the drugs (possibly with some alpha correction), again with the differences, to see which drugs differ.
On the other hand, if what you want is just to find out if post differs from pre for each of the three drugs, but aren't comparing the drugs, you could run the Wilcoxon signed rank test three times (again, possibly with some alpha correction).