Email marketing campaign test design We are designing an email campaign with multiple email touch points. And we are trying to understand the incremental value of each additional touch point. Our experimental design will be that the whole population will be sent the 1st email, then we will randomly split the population 50/50 into our control and testing group. The control group will NOT be sent the 2nd email, while the testing group will receive a second email. I am currently hung up on how I can read the result so I can present on the INCREMENTAL value the 2nd piece of email brings ( what is the value of the second email only).
For example if my entire population is 1000, they will all receive the 1st email, and if we assume open rate is 20%. Then
1st open : 200
1st unopen: 800
Then the 1000 population will be randomly split 50/50 for the 2nd email. So we would have:
Group 1:1st email opened, 2nd email send: 100
Group 2: 1st email opened, 2nd email not send: 100
Group 3: 1st email not opened, 2nd email send: 400
Group 4: 1st email not opened, 2nd email not send:400
The 2nd email opens will be among those who are sent the 2nd email (group 1 + group 3 = 500)
So in order to measure the incremental value of the 2nd email. To calculate my 2nd open rate, should I be measuring:
Option 1: control (group 4 + group 2) vs test ( group 3 + group 1)
Option 2: control (group 4) vs test(group 3)
Essentially, option 1 is including those who took action from the 1st email while option 2 is excluding those ppl.
 A: I think what you can do is split the population into two groups (people who will only get the first email, and people who will get both) and compare engagement rates between the two.  That sounds simple (which is a good thing) and is not exactly what you have said you want to do here, but I will explain why I think its best rather than comparing people who opened/didn't open the email.
The Intention To Treat (ITT) Principle states you should compare people based on what exposure they were assigned to, not what exposure they complied with.  To this extent, because you're interested in engagement rates of one email versus two, you should analyze the results based on what arm people were assigned to prior to the experiment starting.
This means that your allocation scheme of splitting openers and non-openers into 50/50 is not the best approach.  Instead, you should decide on how many of the second emails you are willing to send out, and randomly assign people to getting either one or two.  The experiment should terminate at some pre-specified date, and the engagement metric can be interpreted as "engagement by X days/weeks/months after contact".
The analysis thereafter is straightforward.  You can use a test of proportions if you want.
A: @SaraH :: Since your objective is to find incremental value of second email. Your approach should be
Email 2 :
From theory perspective you are expecting count in each group to be equally distributed. But in practice that depends on your sampling methodology - In case of simple random sampling proportions might vary while in case of stratified random sampling proportions will be same as you mentioned (100,100,400,400)
Group 1:1st email opened, 2nd email send: 100
Group 2: 1st email opened, 2nd email not send: 100
Group 3: 1st email not opened, 2nd email send: 400
Group 4: 1st email not opened, 2nd email not send:400
Getting back to your question, since your second e-mail will have an impact only on Group 3 and Group 4 (Considering you have mentioned impact of 1 open is same as impact of 2 opens. Your group 1 and group 2 are already influenced. So, whether they open second email or not it has no impact on your metric).
Your TG/CG definition should be
Option 2: control (group 4) vs test(group 3)
