5
$\begingroup$

I have various clinical data on participants in a study. I'm looking at a continuous variable ("A") and a (binary) categorical variable (group) ("O"). I used a Wilcoxon test in R (the data are not normally distributed) to see if "A" is significantly different between the two groups. I got a borderline p-value of 0.054.

If I run the Wilcoxon again but include only the males (30 of 72), the p-value is ~0.3; for females only it's ~0.25.

How is it possible that there is no difference in "A" between the groups for males and females separately, but when combined there is a difference?

|cite|improve this question
$\endgroup$
7
$\begingroup$

It seems to be a question of test power. If you only look at a subset you have a lot less participants and therefore a lot less power to find an effect of similar size.

With a reduced sample size you can only find a much bigger effect. So it is NOT recommended to only look at the subsets in this case. Unless there is an interaction (i.e., do the results point into the same direction for both men and women?).

Furthermore, there is no need to use a Wilcoxon test only because your data is not normally distributed (unless it heavily deviates). Probably you can still use the t.test (for example one of the user here, whuber, recently advocated the t.test in a similar case, because the normally assumption does not necessarily hold for the data but for the sampling distribution. quoting him: "The reason is that the sampling distributions of the means are approximately normal, even though the distributions of the data are not").

However, if you still don't want to use the t.test there are more powerful 'assumption free' parametric alternatives, especially permutation tests. See the answer to my question here (whubers quote is also from there): Which permutation test implementation in R to use instead of t-tests (paired and non-paired)?
In my case the results were even a little bit better (i.e., smaller p) than when using the t.test. So I would recommend this permutation test based on the coin package. I could provide you with the necessary r-commands if you provide some sample data in your question.

Update: The effect of outliers on the t-test
If you look at the help of t.test in R ?t.test, you will find the following example:

t.test(1:10,y=c(7:20))      # P = .00001855
t.test(1:10,y=c(7:20, 200)) # P = .1245    -- NOT significant anymore

Although in the second case you have a much extremer difference in the means, the outlier leads to the counterintuitive finding that the data is not significant anymore. Hence, a method to deal with outliers (e.g. Winsorizing, here) is recommended for parametric tests as the t if the data permits.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thanks for the answer and the links. I can't up-vote it yet :) I've tried the t-test on some of the data. The problem is that in some cases Wilcoxon and t-test give the same/similar results, but in other cases (as above), Wilcoxon gives a p-value ~ 0.05 and t-test gives a p-value ~ 0.5. Some of my data contain (a few) outliers. $\endgroup$ – SabreWolfy Jan 18 '11 at 13:51
  • $\begingroup$ @SabreWolfy But you can accept it by clicking on the check-mark icon. $\endgroup$ – user88 Jan 18 '11 at 13:58
  • $\begingroup$ @Henrik I beg to differ: I do not "always advocate the t-test." In one particular situation, because the sample sizes were sufficiently large, the data did not appear to be extremely skewed, the data values were restricted to a finite set, and the t-test was likely more conservative than a permutation test (like the Wilcoxon), I argued that the t-test would perform well. $\endgroup$ – whuber♦ Jan 18 '11 at 14:02
  • 1
    $\begingroup$ @Henrik Your response seems to assume, implicitly, that the Wilcoxon test is not a permutation test. However, both Wilcoxon tests (the signed rank and rank sum tests) are permutation tests. What other permutation tests are you specifically recommending here? $\endgroup$ – whuber♦ Jan 18 '11 at 14:03
  • 1
    $\begingroup$ @SabreWolfy Outliers are always critical to parametric tests (e.g., the t.test). One way of dealing with them, besides the permutation test I highly recommend, is Winsorizing (see e.g.,: unt.edu/rss/class/mike/5700/articles/robustAmerPsyc.pdf). However, with clinical data, outliers can be meaningful. This depends on what the outcome variable is (e.g., a single dead patient is pretty meaningful). No general solution to this problem. $\endgroup$ – Henrik Jan 18 '11 at 14:07
6
$\begingroup$

This is not necessarily an issue of statistical power; it could also be an example of confounding.

Example:

  • One category of $O$ is more common in males but the other is more common in females
  • The distribution of $A$ differs between males and females
  • Within each sex separately, the distribution of $A$ is exactly the same for both $O$ categories

Then there will still be an overall difference in the distribution of $A$ between the $O$ categories.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thanks for pointing these out; I hadn't specifically considered these. $\endgroup$ – SabreWolfy Jan 18 '11 at 14:46
  • $\begingroup$ (+1) I was about to post a response with Simpson's paradox. $\endgroup$ – chl Jan 18 '11 at 15:02
  • $\begingroup$ Just when I think that I've "completed" this analysis, I realize there is more to consider... :) $\endgroup$ – SabreWolfy Jan 18 '11 at 15:06
  • $\begingroup$ You can never complete an analysis! you can only throw your hands in the air and say "I'm sick of this problem, and I accept the errors that will come from not digging deeper!" :) $\endgroup$ – probabilityislogic Jan 21 '11 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.