Statistical testing of proportions to include multiple experiments

Question

I am struggling with understanding what is the correct statistical method to use to determine the statistical difference in proportions for multiple experiments. Let me explain my scenario.

I have run 3 advertisement experiments on 3 different days (which is represented by 3 experiments below. Please assume apart from sample size every other characteristics are same across each day such as gender, user profile etc.).

Experiment 1

	Clicked on Ad	Did not click on Ad
Control	100	900
Test	12	88

Experiment 2

	Clicked on Ad	Did not click on Ad
Control	45	500
Test	14	80

Experiment 3

	Clicked on Ad	Did not click on Ad
Control	250	1500
Test	100	900

The proportion I am interested in is click through rate (CTR). CTR = number of users who clicked on ad/total number of users who saw the ad. For example, for experiment 1 for control group the CTR = 100/(100+900) = 0.1 or 10%. So I want to know if my test variant has better (statistically significant) CTR than my control variant or not?

So there are two approaches I can think of test statistical significance,

1. For each experiment (or day) determine CTR's for both control and tests. So now I have 3 different values of CTRs for controls and tests. Calculate means of the CTRs of both groups and run t-test to determine difference in means.

2. Sum all the controls and tests group across experiments to create one contigency table and than run a chi-squared test on them to determine difference in proportions.

I dont know if either one of this approach is correct or is there an another approach which is correct.

Again, my goal is to know if my test variant has statistically different CTR than my control variant?

Thank you for the help,

Answer 1

Because you're willing to make the assumption that the subjects are the same from day to day (and assuming there are no effects of day that the experiment was conducted) then you can pool the data into a single 2x2 table and do a test of proportions.

in R

ex1 = matrix(c(100,12, 900, 88), nrow = 2)
ex2 = matrix(c(45,14,500,80), nrow = 2)
ex3 = matrix(c(250,100,1500,900), nrow = 2)

total = ex1 + ex2 + ex3

colnames(total) = c('clicked','no click')
rownames(total) = c('control','test')
total

       clicked no click
control     395     2900
test        126     1068

prop.test(x=c(126, 395), n = c(126 + 1068,2900 + 395))

    2-sample test for equality of proportions
    with continuity correction

data:  c(126, 395) out of c(126 + 1068, 2900 + 395)
X-squared = 1.6223, df = 1, p-value =
0.2028
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.035577991  0.006876059
sample estimates:
   prop 1    prop 2 
0.1055276 0.1198786 

Looks like you fail to reject the null. The data do not support a difference in click through rate.