I'm running a Chi Square test in SPSS to see if there are differences in the variable sex (levels: male, female) between my three treatment groups (levels: CBT, DBT, and no treatment).
When I run the Chi Square test, I get a non-significant result (p= ~.14) with a small effect size. This seems a bit odd to me because it seems clear that the m:f ratios in at least two of the groups is quite different... In CBT group I have 19 males and 18 females; In DBT group I have something like 7 males and 17 females; and in no treatment group I have something like 9 males and 18 females.
So my first question is: how am I getting a non-significant result here?
Given that this seemed strange to me, I decided to just run three separate Chi Square tests, each comparing two of the three groups, to see if something changes.
Test 1 compared DBT vs no treatment group, as this returned a non-significant result and small effect, as expected (given the male:female proportions are very similar in these two groups). Test 2 compared CBT and DBT, and this returned a non-significant result and small effect size... The p value and effect size were very similar to the p value and effect size for the original chi square test I ran that included all 3 groups... So these results match the chi square test I ran that included all 3 groups, but this still seems strange that I would get this result given when just comparing CBT and DBT given the seemingly very different m:f proportions in the two groups. Finally, in test 3 I compared CBT and no treatment group and this returned a trend towards significance (p value was over .05 but less than .08)...
Now this result makes sense to me given the difference in male:female ratio between CBT and no treatment group... But two things seem strange: 1) I got a non-significant result when comparing all 3 groups at once - so how am I getting a significant result when comparing two of those groups? 2) For the CBT vs. DBT comparison I got a non-significant result, and yet the male:female ratio for DBT and no treatment group is relatively similar, and yet comparing DBT to CBT returned a non-significant result, whereas comparing no treatment group to CBT returned a significant result. How is this possible?
Another question then is: since comparing all 3 groups at once vs. making 3 separate comparisons of 2 groups each returned different results which then should I run and report for the paper that I'm writing for this project? I got a significant result when comparing CBT and no treatment group but a non-significant result when comparing all 3 groups at once. And if I should be running the full chi squared test with all three groups and not breaking it apart into 3 separate tests, what are some good ways to still account for the clear differences in male:female ratio when running further statistical tests to test my hypotheses.
For example, I'll be testing whether there are differences in anxiety levels between the three groups using ANOVA - one idea is to control for sex... And another idea is to run separate ANOVAs comparing just males and females across groups (and also within groups). Is there anything else that can be done?