0
$\begingroup$

I am struggling with understanding what is the correct statistical method to use to determine the statistical difference in proportions for multiple experiments. Let me explain my scenario.

I have run 3 advertisement experiments on 3 different days (which is represented by 3 experiments below. Please assume apart from sample size every other characteristics are same across each day such as gender, user profile etc.).

Experiment 1

Clicked on Ad Did not click on Ad
Control 100 900
Test 12 88

Experiment 2

Clicked on Ad Did not click on Ad
Control 45 500
Test 14 80

Experiment 3

Clicked on Ad Did not click on Ad
Control 250 1500
Test 100 900

The proportion I am interested in is click through rate (CTR). CTR = number of users who clicked on ad/total number of users who saw the ad. For example, for experiment 1 for control group the CTR = 100/(100+900) = 0.1 or 10%. So I want to know if my test variant has better (statistically significant) CTR than my control variant or not?

So there are two approaches I can think of test statistical significance,

  1. For each experiment (or day) determine CTR's for both control and tests. So now I have 3 different values of CTRs for controls and tests. Calculate means of the CTRs of both groups and run t-test to determine difference in means.

  2. Sum all the controls and tests group across experiments to create one contigency table and than run a chi-squared test on them to determine difference in proportions.

I dont know if either one of this approach is correct or is there an another approach which is correct.

Again, my goal is to know if my test variant has statistically different CTR than my control variant?

Thank you for the help,

$\endgroup$
5
  • $\begingroup$ So to clarify, across all 3 experiments you're testing the same change? These are different people in each experiment, correct? $\endgroup$ Feb 2 '21 at 23:54
  • $\begingroup$ Yes. Its the same test & control variant. Just different people. $\endgroup$
    – prasadav
    Feb 2 '21 at 23:57
  • $\begingroup$ You're free to pool them into a single 2x2 table and do a test of proportions. $\endgroup$ Feb 2 '21 at 23:57
  • $\begingroup$ Thanks. So in other words, you recommend the second approach is the correct one to run and not the first one. $\endgroup$
    – prasadav
    Feb 2 '21 at 23:59
  • $\begingroup$ Yes. I'm writing an answer now. $\endgroup$ Feb 3 '21 at 0:00
0
$\begingroup$

Because you're willing to make the assumption that the subjects are the same from day to day (and assuming there are no effects of day that the experiment was conducted) then you can pool the data into a single 2x2 table and do a test of proportions.

in R


ex1 = matrix(c(100,12, 900, 88), nrow = 2)
ex2 = matrix(c(45,14,500,80), nrow = 2)
ex3 = matrix(c(250,100,1500,900), nrow = 2)

total = ex1 + ex2 + ex3

colnames(total) = c('clicked','no click')
rownames(total) = c('control','test')
total

       clicked no click
control     395     2900
test        126     1068

prop.test(x=c(126, 395), n = c(126 + 1068,2900 + 395))

    2-sample test for equality of proportions
    with continuity correction

data:  c(126, 395) out of c(126 + 1068, 2900 + 395)
X-squared = 1.6223, df = 1, p-value =
0.2028
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.035577991  0.006876059
sample estimates:
   prop 1    prop 2 
0.1055276 0.1198786 

Looks like you fail to reject the null. The data do not support a difference in click through rate.

$\endgroup$
1
  • $\begingroup$ Thank you for your guidance. Make sense. $\endgroup$
    – prasadav
    Feb 3 '21 at 0:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.