# Chi Square test with massive difference in sample size

Let's say I have a frequency table of two independent groups, structured like so:

Control  0.0    1.0    All
----------------------------
False    3648   2205    5853
True     33480  18132  51612
All      37128  20337  57465


And I want to run an A/B test to see whether the two randomly assigned populations performed a certain action.

Are these two populations so vastly different in sample size that it will mess up the math to see if there is a statistically significant relationship?

• Please give the 2-way table with row and column totals. If 33480 is for Gp A, is it total in Bp A or is it count of subjects in A with 'no action'? How was it determined whether subject is in Gp A or Gp B. Random assignment? // Just tact that Gps A and B are of different sizes is not a problem. Why so different may be. Jan 7, 2021 at 22:50
• Updated the frequency table, and added the margins. The two populations were just randomly assigned.
– Pwon
Jan 7, 2021 at 23:27
• There is assumptions to the chisq regarding relative samples size between groups. I see no problems performing the test. Jan 7, 2021 at 23:42

One possibility is to consider binomial proportions of False responses in the two groups. Is the difference in the observed proportions false (roughly $$0.098$$ and $$0.108,$$ respectively), significantly different between the two groups?

In R, this test (which uses a normal approximation) is done using the prop.test procedure. (I have opted not to use continuity correction on account of the large sample sizes.) The null hypothesis that proportions are equal is strongly rejected with P-value $$0.00012 < .05 = 5\%.$$

 prop.test(c(3648, 2205), c(37128, 20337), cor=F)
2-sample test for equality of proportions
without continuity correction

data:  c(3648, 2205) out of c(37128, 20337)
X-squared = 14.851, df = 1, p-value = 0.0001163
alternative hypothesis: two.sided
95 percent confidence interval:
-0.01540543 -0.00493134
sample estimates:
prop 1     prop 2
0.09825469 0.10842307


An almost-equivalent test is to consider whether a chi-squared test of homogeneity across Groups is rejected. The appropriate $$2\times 2$$ table and results of the test are shown below. Again the null hypothesis of homogeneity between the two groups is strongly rejected.

False = c(3648, 2205);  True = c(33480, 18132)
TBL  = rbind(False, True);  TBL
[,1]  [,2]
False  3648  2205
True  33480 18132

chisq.test(TBL)

Pearson's Chi-squared test
with Yates' continuity correction

data:  TBL
X-squared = 14.74, df = 1, p-value = 0.0001234


Notes: You can look at some of the "Related" pages (right margin) found by our robots for more details. Alternatively, you can look at this NIST page on tests of binomial proportions. Another recent related page.

• Thanks for a detailed response. I use python and used chi2_contingency() to arrive at the second result.
– Pwon
Jan 8, 2021 at 2:27