A/B Test Statistical Analysis - Margin of Error

Question

We ran a number of tests (A/B 50/50 split) and replicated over five days, with five rows of results: (EXAMPLE DATA STRUCTURE)

Test#         COHORT A                      COHORT B
              Deliveries   Opens  Clicks    Deliveries   Opens   Clicks
1.            1000          Ao     Ac        1000         Bo     Bc
2.            1000          Ao     Ac        1000         Bo     Bc
3.            1000          Ao     Ac        1000         Bo     Bc
4.            1000          Ao     Ac        1000         Bo     Bc
5.            1000          Ao     Ac        1000         Bo     Bc

How would I determine the margin of error in the overall change of click through rate for these two cohorts?

I have calculated the CTR for both cohorts and the general delta between the two for each test, but when it comes to the stats portion of this I'm completely lost. Do I take a z-test of the overall deltas? Or the cohorts individually?

The ideal is to apply the margin of error to say "this is statistically significant" or not. Or am I using the wrong approach?

Answer 1

Sorry, in my Comment (now deleted), I think I misinterpreted your 'data structure' example.

With such a brief discussion and not much mention of what issues are important, it is difficult to know how to help. I will make up some data, and show results from Minitab (which tends to have user-friendly output for beginners).

Maybe this will provide your answer or a framework for a revised question and a more helpful answer. We understand if you can't reveal certain proprietary information, but you have to address the main issues before we can be of much help.

For Day 1: In Cohort A, suppose 257 were opened and 77 were clicked. In Cohort B, 271 opened an 95 clicked. Then we can find the CTRs for A and B, and compare them to see if they are significantly different. This is a test 'comparing two binomial proportions'. (For more discussion about such tests, perhaps see this Q&A or this---oe one of the links under 'Related' in the right-hand margin of this page.)

Minitab's version gives the following output:

Test and CI for Two Proportions 

Sample   X    N  Sample p
1       77  256  0.300781
2       95  271  0.350554

Difference = p (1) - p (2)
Estimate for difference:  -0.0497723
95% CI for difference:  (-0.129666, 0.0301218)
Test for difference = 0 (vs ≠ 0):  
   Z = -1.22  P-Value = 0.223

Although cohort A has a CTR of .30 and B a CTR of .35, the numbers opened are not large enough for these two rates to be deemed significantly different at the 5% level: the P-value $0.223 > 0.05.$

You might also wonder whether the rate of opening differed according to Cohort. Then you could use counts 1000 Deliveries and number Opened to run a similar test.

If you seem to get substantially different results (for Clicking opened sites, or for Opening delivered sites) on different days, then you might want to look at a 3-way contingency table with three categorical variables: Day (1-5), Cohort (A, B), and Result (Open, Not). Then you could test whether results really do differ from day to day. If the five days give very similar results, I would consider combining counts for all five days.