1
$\begingroup$

We are designing an email campaign with multiple email touch points. And we are trying to understand the incremental value of each additional touch point. Our experimental design will be that the whole population will be sent the 1st email, then we will randomly split the population 50/50 into our control and testing group. The control group will NOT be sent the 2nd email, while the testing group will receive a second email. I am currently hung up on how I can read the result so I can present on the INCREMENTAL value the 2nd piece of email brings ( what is the value of the second email only).

For example if my entire population is 1000, they will all receive the 1st email, and if we assume open rate is 20%. Then

1st open : 200

1st unopen: 800

Then the 1000 population will be randomly split 50/50 for the 2nd email. So we would have:

Group 1:1st email opened, 2nd email send: 100

Group 2: 1st email opened, 2nd email not send: 100

Group 3: 1st email not opened, 2nd email send: 400

Group 4: 1st email not opened, 2nd email not send:400

The 2nd email opens will be among those who are sent the 2nd email (group 1 + group 3 = 500)

So in order to measure the incremental value of the 2nd email. To calculate my 2nd open rate, should I be measuring:

Option 1: control (group 4 + group 2) vs test ( group 3 + group 1)

Option 2: control (group 4) vs test(group 3)

Essentially, option 1 is including those who took action from the 1st email while option 2 is excluding those ppl.

Improve this question
$\endgroup$
4
  • $\begingroup$ What behaviour are you looking at? If it is opening up the second email, I don't see why you need the second control group (since you can't measure their outcome by definition). Are you looking to use the first email as measure of baseline open rate? $\endgroup$
    – Demetri Pananos
    Mar 10, 2021 at 2:15
  • $\begingroup$ Yes, the first email will be used to establish baseline. Our hypothesis is that the second email will bring additional engagement to the program. That’s why it’s design to be an ab test to measure the incremental value of the second piece. $\endgroup$
    – Sara H
    Mar 10, 2021 at 2:26
  • $\begingroup$ Great. Can you tell me how engagement is being measured (e.g. signups, button clicks, etc)? $\endgroup$
    – Demetri Pananos
    Mar 10, 2021 at 2:27
  • $\begingroup$ We will be measuring clicks of the url $\endgroup$
    – Sara H
    Mar 10, 2021 at 2:34

2 Answers 2

Reset to default
1
$\begingroup$

I think what you can do is split the population into two groups (people who will only get the first email, and people who will get both) and compare engagement rates between the two. That sounds simple (which is a good thing) and is not exactly what you have said you want to do here, but I will explain why I think its best rather than comparing people who opened/didn't open the email.

The Intention To Treat (ITT) Principle states you should compare people based on what exposure they were assigned to, not what exposure they complied with. To this extent, because you're interested in engagement rates of one email versus two, you should analyze the results based on what arm people were assigned to prior to the experiment starting.

This means that your allocation scheme of splitting openers and non-openers into 50/50 is not the best approach. Instead, you should decide on how many of the second emails you are willing to send out, and randomly assign people to getting either one or two. The experiment should terminate at some pre-specified date, and the engagement metric can be interpreted as "engagement by X days/weeks/months after contact".

The analysis thereafter is straightforward. You can use a test of proportions if you want.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thank you Demetri! My initial post might be abit confusing, the split is not based on opens, it's based the original population(1000 ppl). So within the test group (both 1st email and 2nd email send), it could contain open activities from the 1st email or 2nd email or both emails. I wanted to confirm if I'm understanding the 1 email vs. 2 email proportion test. For the 2 email proportion, should I be calculating the open rate using (# opens from 1st email + # opens from 2nd email) / (# of 1st email send + # 2nd email send) or should it be (# opens from 2nd email) / (# 2nd email sends)? $\endgroup$
    – Sara H
    Mar 10, 2021 at 4:03
  • $\begingroup$ @SaraH Good question. You mean to say engagement, not opens right? That's why I initially asked about what behaviour you are tracking. If you're interested in determining if 2 emails drives more engagement rather than 1 email, your metric is # of people who engaged / # people in the group. So in the 2 email group, it won't matter if they engaged after the first or second or after both. Its all counted as a single engagement. $\endgroup$
    – Demetri Pananos
    Mar 10, 2021 at 4:37
  • $\begingroup$ @ DemetriPananos thank you! So for the open activity in the 2 email proportion, regardless of which email they open, and how many times this person opens the email, if an open happens I would be counting this as a single unique open event. And any clicks that happens (regardless of which email this click came from, and how many times they click) I would treat this as a single unique engagement event. I just wanted to confirm again, because the click rate will be calculated based on # of opens. So for this rate = # of ppl with unique click event / # of ppl with unique open event $\endgroup$
    – Sara H
    Mar 10, 2021 at 5:48
  • $\begingroup$ @SaraH Click rate based on number of opens seems dubious. The whole point of the experiment is to see if sending one or two emails makes you more likely to click. That you did not open the email means you did not click. Additionally, you have no control over who opens the emails, so your sample size and hence power will drop dramatically. $\endgroup$
    – Demetri Pananos
    Mar 10, 2021 at 14:20
  • $\begingroup$ thanks! The clicks to open measurement is useful for us to understand how effective the content of the email is. Whether the content of the email leads to more call to action (clicks on the url). Aside from the drop of power and sample size, would there be other concerns with this methodology? $\endgroup$
    – Sara H
    Mar 10, 2021 at 20:50
0
$\begingroup$

@SaraH :: Since your objective is to find incremental value of second email. Your approach should be

Email 2 :

From theory perspective you are expecting count in each group to be equally distributed. But in practice that depends on your sampling methodology - In case of simple random sampling proportions might vary while in case of stratified random sampling proportions will be same as you mentioned (100,100,400,400)

Group 1:1st email opened, 2nd email send: 100

Group 2: 1st email opened, 2nd email not send: 100

Group 3: 1st email not opened, 2nd email send: 400

Group 4: 1st email not opened, 2nd email not send:400

Getting back to your question, since your second e-mail will have an impact only on Group 3 and Group 4 (Considering you have mentioned impact of 1 open is same as impact of 2 opens. Your group 1 and group 2 are already influenced. So, whether they open second email or not it has no impact on your metric).

Your TG/CG definition should be

Option 2: control (group 4) vs test(group 3)

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.