0
$\begingroup$

I'm running a Chi Square test in SPSS to see if there are differences in the variable sex (levels: male, female) between my three treatment groups (levels: CBT, DBT, and no treatment).

When I run the Chi Square test, I get a non-significant result (p= ~.14) with a small effect size. This seems a bit odd to me because it seems clear that the m:f ratios in at least two of the groups is quite different... In CBT group I have 19 males and 18 females; In DBT group I have something like 7 males and 17 females; and in no treatment group I have something like 9 males and 18 females.

So my first question is: how am I getting a non-significant result here?

Given that this seemed strange to me, I decided to just run three separate Chi Square tests, each comparing two of the three groups, to see if something changes.

Test 1 compared DBT vs no treatment group, as this returned a non-significant result and small effect, as expected (given the male:female proportions are very similar in these two groups). Test 2 compared CBT and DBT, and this returned a non-significant result and small effect size... The p value and effect size were very similar to the p value and effect size for the original chi square test I ran that included all 3 groups... So these results match the chi square test I ran that included all 3 groups, but this still seems strange that I would get this result given when just comparing CBT and DBT given the seemingly very different m:f proportions in the two groups. Finally, in test 3 I compared CBT and no treatment group and this returned a trend towards significance (p value was over .05 but less than .08)...

Now this result makes sense to me given the difference in male:female ratio between CBT and no treatment group... But two things seem strange: 1) I got a non-significant result when comparing all 3 groups at once - so how am I getting a significant result when comparing two of those groups? 2) For the CBT vs. DBT comparison I got a non-significant result, and yet the male:female ratio for DBT and no treatment group is relatively similar, and yet comparing DBT to CBT returned a non-significant result, whereas comparing no treatment group to CBT returned a significant result. How is this possible?

Another question then is: since comparing all 3 groups at once vs. making 3 separate comparisons of 2 groups each returned different results which then should I run and report for the paper that I'm writing for this project? I got a significant result when comparing CBT and no treatment group but a non-significant result when comparing all 3 groups at once. And if I should be running the full chi squared test with all three groups and not breaking it apart into 3 separate tests, what are some good ways to still account for the clear differences in male:female ratio when running further statistical tests to test my hypotheses.

For example, I'll be testing whether there are differences in anxiety levels between the three groups using ANOVA - one idea is to control for sex... And another idea is to run separate ANOVAs comparing just males and females across groups (and also within groups). Is there anything else that can be done?

$\endgroup$
7
  • 3
    $\begingroup$ You have small counts. The apparent differences in relative proportions can plausibly be explained by random variation. Efforts to run additional tests upon observing your planned test(s) are not significant are known as "post hoc testing" and are useful, at best, to make suggestions for further studies. A good keyword search is "HARKing", which refers to situations where the course of your development of hypotheses and their testing is not appropriately reported. $\endgroup$
    – whuber
    Feb 17, 2023 at 20:22
  • 1
    $\begingroup$ In addition to the other comment and answer (+1), avoid reporting a "trend towards significance", this is considered misleading: bmj.com/content/348/bmj.g2215 . $\endgroup$
    – J-J-J
    Feb 17, 2023 at 20:25
  • 1
    $\begingroup$ Thanks to you both. I know about post-hoc testing... But I never really thought of it in the context of chi square tests before somehow... I will check out HARKing. $\endgroup$ Feb 17, 2023 at 20:39
  • 3
    $\begingroup$ When samples are small, the standard error of the difference in proportion is large, so you need a very large difference in sample proportion to get something not explainable as no difference in population proportion + random variation. There's nothing really to be done about that; small samples are small (it's easy to use simulation to show yourself that really large differences in sample proportion can happen when the population proportions are equal and sample sizes are small). Plan all your tests at the start (before you see data), or your p-values no longer mean what you want them to. $\endgroup$
    – Glen_b
    Feb 17, 2023 at 23:02
  • 1
    $\begingroup$ I think some of the confusion also stems from placing too much emphasis on p-value cutoff of 0.05. Getting a p-value of 0.14 on an omnibus test, and then a p-value of 0.07 on what might be thought of as a post-hoc test isn't too unusual. The two p-values aren't that different. The fact that 0.07 is close to 0.05 relies on us putting a lot of importance on that 0.05 cutoff. If our cutoff were 0.01, we wouldn't be thinking that a p-value of 0.14 and 0.07 have potentially different implications. $\endgroup$ Feb 18, 2023 at 17:22

1 Answer 1

5
$\begingroup$

Basically your data are:

\begin{array}{c|ccc} & DBT & CBT & No tx\\ \hline M & 7 & 19 & 9 \\ F & 17 & 18 & 18 \\ \end{array}

Although "$\chi^2$ tests" tend to be agnostic about exposure versus outcome, you speak of the results as if you're modeling gender as a function of treatment and not vice versa as we would strongly suspect.

I can confirm that these results are not significant, although my $p$-value is different, which may be a typo or transcription error, but in either case this shouldn't be surprising. I'm a bit surprised by what you are declaring to be "effect size", since you might mean test statistic.

The Chi-square test statistic is $$ T = \sum (\frac{\text{(Observed - Expected)}^2}{\text{Expected}})$$

What's "expected" is highly dependent on the test. Therefore it's not specific to call it a "Chi-square" test since almost every statistical test statistic out there can be compared to a Chi-square distribution under the null. The Pearson chi-square test of homogeneity or of independence is usually what we mean. In this case, the expected proportion of males in each group would be 100 * (7 + 19 + 9) / (17+18+18+7+19+9) = 39.7%. Then given there are 24 in DBT, 37 in CBT and 27 in notx, we'd expect the following as a table of "expected" frequencies

\begin{array}{c|ccc} & DBT & CBT & No tx\\ \hline M & 9.5 & 14.7 & 10.7 \\ F & 14.5 & 22.3 & 16.2 \\ \end{array}

which calculating the $T$ as above gives a meager 3.79 which is shy of the critical value corresponding to a 0.05 test. In short, statistician and non-statistician alike, our intuition often falls short of reality when it comes to the necessary sample size to render a significant effect for categorical data analyses.

To your second question, it's clear based on your description that the primary inferential comparison was the overall Pearson test that I described above. Reporting anything different as a "main" analysis would be severely misleading. As far as "post-hoc" or "exploratory" analyses, well sky's the limit, but ethical considerations hold you to be honest with the community about what the primary outcome and primary result are.

$\endgroup$
6
  • $\begingroup$ Reason why you probably got a different p value: I actually had the # of subject in each group a little bit off in the original post because I forgot. I changed them to the actual #s. Thanks for your reply... I think what you're saying makes sense... but then why is it that I'm getting a significant result when I run three chi square tests instead of one? (recall in my post I say that I say that when I ran CBT vs no treatment I got a nearly significant result... but when I ran all three groups at once it was non-significant)... Not totally clear to me why that is.. $\endgroup$ Feb 17, 2023 at 20:30
  • $\begingroup$ Addendum to above comment: I meant to say that I'm getting a nearly significant result when I run 3 chi square tests instead of one $\endgroup$ Feb 17, 2023 at 20:40
  • $\begingroup$ Again, I understand why the overall test might be non-significant... I just don't quite understand the mismatch between the overall test and the post-hoc one... $\endgroup$ Feb 17, 2023 at 20:42
  • $\begingroup$ Also, I was referring to effect size, not the chi square statistic... I had SPSS calculate Cramer's V $\endgroup$ Feb 17, 2023 at 20:47
  • 3
    $\begingroup$ @FastBallooningHead the discrepancies between the "overall" test and separate two-way tests will usually boil down to the heterogeneity of groups and of sample sizes. Cherry picking the most disparate groups may give a test that is more significant than the overall test. This is because the degrees of freedom are fewer. A Chisq test has degrees of freedom given by: (number of rows - 1) * (number of columns - 1). In your case, 2 vs 1 $\endgroup$
    – AdamO
    Feb 17, 2023 at 22:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.