We are designing an email campaign with multiple email touch points. And we are trying to understand the incremental value of each additional touch point. Our experimental design will be that the whole population will be sent the 1st email, then we will randomly split the population 50/50 into our control and testing group. The control group will NOT be sent the 2nd email, while the testing group will receive a second email. I am currently hung up on how I can read the result so I can present on the INCREMENTAL value the 2nd piece of email brings ( what is the value of the second email only).
For example if my entire population is 1000, they will all receive the 1st email, and if we assume open rate is 20%. Then
1st open : 200
1st unopen: 800
Then the 1000 population will be randomly split 50/50 for the 2nd email. So we would have:
Group 1:1st email opened, 2nd email send: 100
Group 2: 1st email opened, 2nd email not send: 100
Group 3: 1st email not opened, 2nd email send: 400
Group 4: 1st email not opened, 2nd email not send:400
The 2nd email opens will be among those who are sent the 2nd email (group 1 + group 3 = 500)
So in order to measure the incremental value of the 2nd email. To calculate my 2nd open rate, should I be measuring:
Option 1: control (group 4 + group 2) vs test ( group 3 + group 1)
Option 2: control (group 4) vs test(group 3)
Essentially, option 1 is including those who took action from the 1st email while option 2 is excluding those ppl.