Chi-square independence test for A/B split testing

Question

I have the results of an A/B split test of a web page.

Page  Impressions   Clicks    CTR       
A     56,000        10        0.018%    
B     78,000        21        0.027%    

I'd like to test the ctr for statistically significant difference. Null hypothesis - there is no difference in CTR between pages A & B.

How would I go about doing this on a more theoretical level? As in what should be rows, column, setting expected values?

Answer 1

Here's one way to lay out the chi-squared test -- you don't necessarily need all this detail. The test doesn't care which way around you have rows and columns, so I'll do it your way around.

Comparing proportions via Pearson's Chi-squared test of independence
at the 5% level      #(for this example, at least - you choose your own level)

Null hypothesis - there is no difference in CTR between pages A & B.

Observed:
   Clicks   Nonclicks     Impressions 
A     10       55990           56000
B     21       77979           78000

Tot   31      133969          134000

Expected:
   Clicks   Nonclicks     Impressions 
A   12.96    55987.04          56000
B   18.04    77981.96          78000

Tot   31      133969          134000

(Entries in the body of the Expected table are row.total*column.total/overall.total)

Contribution to chi-square = (Observed - Expected)^2/Expected

       Clicks   Nonclicks   
 A     0.6741   0.0001560
 B     0.4840   0.0001120

Chi-square = sum((Observed - Expected)^2/Expected)
           = 1.158368

 df = 1
 p-value = 0.2818  

 At the 5% level we do not reject H0 - there's no evidence of a difference in
 click through rate.

(You need tables or a program to find the p-value.)

Answer 2

The chi-square test of independence only tests for whether there is a relationship between the variables, though, not that there is no difference between CTR:

H0: Click-through and interface are independent. Ha: Click-through and interface are not independent (that is, something interesting is going on; one of your interfaces is performing better)

You would prepare the test like this:

success <- c(10,21)
failure <- c(55990,77979)
my.table <- rbind(success,failure)

And then run the test:

> chisq.test(my.table)

    Pearson's Chi-squared test with Yates' continuity correction

data:  my.table
X-squared = 0.79955, df = 1, p-value = 0.3712

You get the same results as the previous respondent when you turn off the continuity correction (which many argue you don't need at all, unless you have less than 5 clicks on either A or B):

> chisq.test(my.table, correct=FALSE)

    Pearson's Chi-squared test

data:  my.table
X-squared = 1.1584, df = 1, p-value = 0.2818

You can easily access your expected value table and compute the chi-square test statistic manually if you want:

> chisq.test(my.table, correct=FALSE)$expected
               [,1]        [,2]
success    12.95522    18.04478
failure 55987.04478 77981.95522

An alternative would be the two-proportion zitest, which says:

H0: CTR of B - CTR of A = 0 Ha: CTR of B - CTR of A > 0 (B has a bigger CTR)

> source("https://raw.githubusercontent.com/NicoleRadziwill/R-Functions/master/z2test.R")
> z2.test(10,55990,21,77979)
$estimate
[1] -9.069995e-05

$ts.z
[1] -1.076516

$p.val
[1] 0.1408483

$cint
[1] -2.504372e-04  6.903728e-05

Notice that the p-value is large (0.14) and the confidence interval includes the value zero -- there's no difference between your A and B CTRs.

You could also use a chi-square test statistic to run the test and get a confidence interval:

> prop.test(c(10,21),c(55990,77979))

    2-sample test for equality of proportions with continuity correction

data:  c(10, 21) out of c(55990, 77979)
X-squared = 0.79999, df = 1, p-value = 0.3711
alternative hypothesis: two.sided
95 percent confidence interval:
 -2.657756e-04  8.437566e-05
sample estimates:
      prop 1       prop 2 
0.0001786033 0.0002693033

Notice that the p-value is the same as when you ran the chi-square test of independence earlier, and the confidence interval also includes zero in it.... indicating no significant difference between CTRs for your A and B.