Weird results from Chi Square test

I'm running a chi square test on some categorical values pertaining to race, and whether different racial groups participated in a clinic. As there's about a dozen different races in this data, I bucketed them down to 'White', 'Black' and 'Other', just for the purposes of testing (as the correlations indicated most of the activity occurring between 'White and 'Black'. However, using Python's .chi2_contingency() method, I'm getting results back that seem unusual. The table is below:

    Appointment Status    No    Yes

Black                    9170   33372
White                    15137  152307
Other                    8864   56165

The Python method returns the following:

X^2: 5207.16
p-value: 0.0
df: 2
expected values array:  array([[  5131.21350472,  37410.78649528],
[  7843.48838791,  57185.51161209],
[ 20196.29810738, 147247.70189262]]))


The df is good, but the chi square value and p-value both don't seem right. Is there something anyone can see that I might be doing methodologically that might be producing these values, or might there be something going on behind the scenes in Python that's doing this? Thanks!

• What about these results seems not "right"? The output shows enormous discrepancies between the expected and observed values (even after allowing for its mysterious permutation of the rows). The large $\chi^2$ statistic reflects that and, of course, the p-value ought to be astronomically low. The bottom line here is that chi-squared testing is useless with such large datasets.
– whuber
Commented Sep 30, 2022 at 19:19
• @whuber I suppose that tracks. When I exclude 'Other' from the cross tab, I get an X^2 of 5189, p-value of 0.0 and df of 1, so not much change. I don't know if they're really that not correlated, or if it's just gibberish. Any suggestions on a more appropriate test? Commented Sep 30, 2022 at 20:19
• I wouldn't bother to test at all--it's unnecessary when the differences are this clear--but I can't recommend anything because you haven't explained what your objectives are.
– whuber
Commented Sep 30, 2022 at 22:06
• @whuber Basically, I'm trying to establish correlations (if they exist), between different columns of nominal data (i.e., race, sex, clinic type, etc.) and whether someone shows up to an appointment or not. The overall dataset has an n=286,583, so it would seem that chi square might be gagging on the large amount of data for certain comparisons. Commented Oct 3, 2022 at 13:41
• The test has no problems. It just not useful. What would be a true surprise would be if this test did not detect a difference in such a large dataset!
– whuber
Commented Oct 3, 2022 at 14:49

The chi-square and p values are correct for this contingency table.

To compare, you can run the following in R or at the following website without installing software: rdrr.io/snippets/ .

Matrix = matrix (c(9170, 33372, 15137, 152307, 8864, 56165), byrow=TRUE, nrow=3)

chisq.test(Matrix)

###  Pearson's Chi-squared test
###
### X-squared = 5207.2, df = 2, p-value < 2.2e-16


It's not unusual to get large chi-square and small p values when the sample size is relatively large.

Probably you want some way to assess the effect size, which may be more meaningful than the p-value.

Because you have a dichotomous response (Yes / No), probably the most intuitive way to express this is to present or plot the proportion of Yes responses for each Appointment status. Here, the following R code will return the proportions in each row of the table:

 prop.table(Matrix, margin=1)


Here, the proportions of Yes answers in each row is 0.78, 0.91, 0.86.

Whether or not these differences in Yes responses are meaningful for your application is up to you.

As a side note, sometimes Cramer's V (Cramér's V)is used as measure of effect size for contingency tables. The result of 0.138 is relatively small, but, again, how meaningful this effect size is depends on your judgement.

library(DescTools)

CramerV(Matrix)

### 0.138