Chi Square test with massive difference in sample size

Question

Let's say I have a frequency table of two independent groups, structured like so:

Control  0.0    1.0    All
----------------------------
False    3648   2205    5853
True     33480  18132  51612
All      37128  20337  57465

And I want to run an A/B test to see whether the two randomly assigned populations performed a certain action.

Are these two populations so vastly different in sample size that it will mess up the math to see if there is a statistically significant relationship?

Answer 1

One possibility is to consider binomial proportions of False responses in the two groups. Is the difference in the observed proportions false (roughly $0.098$ and $0.108,$ respectively), significantly different between the two groups?

In R, this test (which uses a normal approximation) is done using the prop.test procedure. (I have opted not to use continuity correction on account of the large sample sizes.) The null hypothesis that proportions are equal is strongly rejected with P-value $0.00012 < .05 = 5\%.$

 prop.test(c(3648, 2205), c(37128, 20337), cor=F)
      2-sample test for equality of proportions 
      without continuity correction

data:  c(3648, 2205) out of c(37128, 20337)
X-squared = 14.851, df = 1, p-value = 0.0001163
alternative hypothesis: two.sided
95 percent confidence interval:
  -0.01540543 -0.00493134
sample estimates:
    prop 1     prop 2 
0.09825469 0.10842307 

An almost-equivalent test is to consider whether a chi-squared test of homogeneity across Groups is rejected. The appropriate $2\times 2$ table and results of the test are shown below. Again the null hypothesis of homogeneity between the two groups is strongly rejected.

False = c(3648, 2205);  True = c(33480, 18132)
TBL  = rbind(False, True);  TBL
       [,1]  [,2]
False  3648  2205
True  33480 18132

chisq.test(TBL)

        Pearson's Chi-squared test 
        with Yates' continuity correction

data:  TBL
X-squared = 14.74, df = 1, p-value = 0.0001234

Notes: You can look at some of the "Related" pages (right margin) found by our robots for more details. Alternatively, you can look at this NIST page on tests of binomial proportions. Another recent related page.