# statistical testing of proportions to include multiple experiments

I am struggling with understanding what is the correct statistical method to use to determine the statistical difference in proportions for multiple experiments. Let me explain my scenario.

I have run 3 advertisement experiments on 3 different days (which is represented by 3 experiments below. Please assume apart from sample size every other characteristics are same across each day such as gender, user profile etc.).

Experiment 1

Control 100 900
Test 12 88

Experiment 2

Control 45 500
Test 14 80

Experiment 3

Control 250 1500
Test 100 900

The proportion I am interested in is click through rate (CTR). CTR = number of users who clicked on ad/total number of users who saw the ad. For example, for experiment 1 for control group the CTR = 100/(100+900) = 0.1 or 10%. So I want to know if my test variant has better (statistically significant) CTR than my control variant or not?

So there are two approaches I can think of test statistical significance,

1. For each experiment (or day) determine CTR's for both control and tests. So now I have 3 different values of CTRs for controls and tests. Calculate means of the CTRs of both groups and run t-test to determine difference in means.

2. Sum all the controls and tests group across experiments to create one contigency table and than run a chi-squared test on them to determine difference in proportions.

I dont know if either one of this approach is correct or is there an another approach which is correct.

Again, my goal is to know if my test variant has statistically different CTR than my control variant?

Thank you for the help,

• So to clarify, across all 3 experiments you're testing the same change? These are different people in each experiment, correct? Feb 2, 2021 at 23:54
• Yes. Its the same test & control variant. Just different people. Feb 2, 2021 at 23:57
• You're free to pool them into a single 2x2 table and do a test of proportions. Feb 2, 2021 at 23:57
• Thanks. So in other words, you recommend the second approach is the correct one to run and not the first one. Feb 2, 2021 at 23:59
• Yes. I'm writing an answer now. Feb 3, 2021 at 0:00

Because you're willing to make the assumption that the subjects are the same from day to day (and assuming there are no effects of day that the experiment was conducted) then you can pool the data into a single 2x2 table and do a test of proportions.

in R


ex1 = matrix(c(100,12, 900, 88), nrow = 2)
ex2 = matrix(c(45,14,500,80), nrow = 2)
ex3 = matrix(c(250,100,1500,900), nrow = 2)

total = ex1 + ex2 + ex3

colnames(total) = c('clicked','no click')
rownames(total) = c('control','test')
total

clicked no click
control     395     2900
test        126     1068

prop.test(x=c(126, 395), n = c(126 + 1068,2900 + 395))

2-sample test for equality of proportions
with continuity correction

data:  c(126, 395) out of c(126 + 1068, 2900 + 395)
X-squared = 1.6223, df = 1, p-value =
0.2028
alternative hypothesis: two.sided
95 percent confidence interval:
-0.035577991  0.006876059
sample estimates:
prop 1    prop 2
0.1055276 0.1198786



Looks like you fail to reject the null. The data do not support a difference in click through rate.

• Thank you for your guidance. Make sense. Feb 3, 2021 at 0:07