I am struggling with understanding what is the correct statistical method to use to determine the statistical difference in proportions for multiple experiments. Let me explain my scenario.

I have run 3 advertisement experiments on 3 different days (which is represented by 3 experiments below. Please assume apart from sample size every other characteristics are same across each day such as gender, user profile etc.).

Experiment 1

Clicked on Ad Did not click on Ad
Control 100 900
Test 12 88

Experiment 2

Clicked on Ad Did not click on Ad
Control 45 500
Test 14 80

Experiment 3

Clicked on Ad Did not click on Ad
Control 250 1500
Test 100 900

The proportion I am interested in is click through rate (CTR). CTR = number of users who clicked on ad/total number of users who saw the ad. For example, for experiment 1 for control group the CTR = 100/(100+900) = 0.1 or 10%. So I want to know if my test variant has better (statistically significant) CTR than my control variant or not?

So there are two approaches I can think of test statistical significance,

  1. For each experiment (or day) determine CTR's for both control and tests. So now I have 3 different values of CTRs for controls and tests. Calculate means of the CTRs of both groups and run t-test to determine difference in means.

  2. Sum all the controls and tests group across experiments to create one contigency table and than run a chi-squared test on them to determine difference in proportions.

I dont know if either one of this approach is correct or is there an another approach which is correct.

Again, my goal is to know if my test variant has statistically different CTR than my control variant?

Thank you for the help,

  • $\begingroup$ So to clarify, across all 3 experiments you're testing the same change? These are different people in each experiment, correct? $\endgroup$ Commented Feb 2, 2021 at 23:54
  • $\begingroup$ Yes. Its the same test & control variant. Just different people. $\endgroup$
    – prasadav
    Commented Feb 2, 2021 at 23:57
  • $\begingroup$ You're free to pool them into a single 2x2 table and do a test of proportions. $\endgroup$ Commented Feb 2, 2021 at 23:57
  • $\begingroup$ Thanks. So in other words, you recommend the second approach is the correct one to run and not the first one. $\endgroup$
    – prasadav
    Commented Feb 2, 2021 at 23:59
  • $\begingroup$ Yes. I'm writing an answer now. $\endgroup$ Commented Feb 3, 2021 at 0:00

1 Answer 1


Because you're willing to make the assumption that the subjects are the same from day to day (and assuming there are no effects of day that the experiment was conducted) then you can pool the data into a single 2x2 table and do a test of proportions.

in R

ex1 = matrix(c(100,12, 900, 88), nrow = 2)
ex2 = matrix(c(45,14,500,80), nrow = 2)
ex3 = matrix(c(250,100,1500,900), nrow = 2)

total = ex1 + ex2 + ex3

colnames(total) = c('clicked','no click')
rownames(total) = c('control','test')

       clicked no click
control     395     2900
test        126     1068

prop.test(x=c(126, 395), n = c(126 + 1068,2900 + 395))

    2-sample test for equality of proportions
    with continuity correction

data:  c(126, 395) out of c(126 + 1068, 2900 + 395)
X-squared = 1.6223, df = 1, p-value =
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.035577991  0.006876059
sample estimates:
   prop 1    prop 2 
0.1055276 0.1198786 

Looks like you fail to reject the null. The data do not support a difference in click through rate.

  • $\begingroup$ Thank you for your guidance. Make sense. $\endgroup$
    – prasadav
    Commented Feb 3, 2021 at 0:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.