1
$\begingroup$

Let's say I have a frequency table of two independent groups, structured like so:

Control  0.0    1.0    All
----------------------------
False    3648   2205    5853
True     33480  18132  51612
All      37128  20337  57465

And I want to run an A/B test to see whether the two randomly assigned populations performed a certain action.

Are these two populations so vastly different in sample size that it will mess up the math to see if there is a statistically significant relationship?

$\endgroup$
3
  • $\begingroup$ Please give the 2-way table with row and column totals. If 33480 is for Gp A, is it total in Bp A or is it count of subjects in A with 'no action'? How was it determined whether subject is in Gp A or Gp B. Random assignment? // Just tact that Gps A and B are of different sizes is not a problem. Why so different may be. $\endgroup$
    – BruceET
    Jan 7, 2021 at 22:50
  • $\begingroup$ Updated the frequency table, and added the margins. The two populations were just randomly assigned. $\endgroup$
    – Pwon
    Jan 7, 2021 at 23:27
  • $\begingroup$ There is assumptions to the chisq regarding relative samples size between groups. I see no problems performing the test. $\endgroup$
    – Dave2e
    Jan 7, 2021 at 23:42

1 Answer 1

0
$\begingroup$

One possibility is to consider binomial proportions of False responses in the two groups. Is the difference in the observed proportions false (roughly $0.098$ and $0.108,$ respectively), significantly different between the two groups?

In R, this test (which uses a normal approximation) is done using the prop.test procedure. (I have opted not to use continuity correction on account of the large sample sizes.) The null hypothesis that proportions are equal is strongly rejected with P-value $0.00012 < .05 = 5\%.$

 prop.test(c(3648, 2205), c(37128, 20337), cor=F)
      2-sample test for equality of proportions 
      without continuity correction

data:  c(3648, 2205) out of c(37128, 20337)
X-squared = 14.851, df = 1, p-value = 0.0001163
alternative hypothesis: two.sided
95 percent confidence interval:
  -0.01540543 -0.00493134
sample estimates:
    prop 1     prop 2 
0.09825469 0.10842307 

An almost-equivalent test is to consider whether a chi-squared test of homogeneity across Groups is rejected. The appropriate $2\times 2$ table and results of the test are shown below. Again the null hypothesis of homogeneity between the two groups is strongly rejected.

False = c(3648, 2205);  True = c(33480, 18132)
TBL  = rbind(False, True);  TBL
       [,1]  [,2]
False  3648  2205
True  33480 18132

chisq.test(TBL)

        Pearson's Chi-squared test 
        with Yates' continuity correction

data:  TBL
X-squared = 14.74, df = 1, p-value = 0.0001234

Notes: You can look at some of the "Related" pages (right margin) found by our robots for more details. Alternatively, you can look at this NIST page on tests of binomial proportions. Another recent related page.

$\endgroup$
1
  • 1
    $\begingroup$ Thanks for a detailed response. I use python and used chi2_contingency() to arrive at the second result. $\endgroup$
    – Pwon
    Jan 8, 2021 at 2:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.