A/B test: measure likelihood of outcomes based on historical events

Question

I am researching if it makes sense to create personalized emails based on past purchase behavior, but I do not know how to design A/B tests. The starting data I have is a list of about 500,000 subscribed email addresses, about 1,700 of which have a common attribute: they've purchased a certain category of product. To do an A/B test, I am considering two options:

1. Let $A$ be the 500,000 general list
2. Let $B$ be the 1,700 specialized list
3. Let $E_1$ be an email template
4. Let $E_2$ be a potential second email template

Question:

• Does it make sense to send use one email template, and when sending it, measure and compare engagement of $A$ and $B$?
• Does it make sense to design two email templates, and for each email template, send half of the total emails sent to $A$ and half to $B$? And see how each group engages with each email template?

In other words I am consider two strategies:

1. Send 2000 emails using a single template, 1000 to $A$, 1000 to $B$; compare engagement rates between the groups on the single template; or
2. Create template $E_1$ and send 500 to $A$ and 500 to $B$ using it; create template $E_2$ and send 500 to $A$ and send 500 to $B$. Measure engagement rates for each template and group, and compare.

To follow up

• Are there advantages or disadvantages to either approach?
• Does one make more sense than the other depending on some factor or goal?
• What methods seem appropriate to approximate the significance of any results?

I am considering measuring engagement based simply on how many people click on a link in the email, or perhaps on some event happening on my website such as a product view, a purchase, or a post in the forums.

My experience with statistics is rather minimal. I am in an intro probability theory and statistics class, but we haven't gotten as far as experiment design yet.

Any advice on more helpful details I can include in my question, or alternative approaches to my experiment would be appreciated. Any links to resources on sorting this out would also be helpful, leaning towards an audience with limited statistical background, though having a willingness to learn. (I have some basic probability theory understanding.)

An overall goal is to determine if the effort of gathering this data and transforming it into something usable by our systems is worth it. Perhaps it will take multiple tests to be confident in this approach.

Answer 1

If the treatment impacts only a tiny fraction of users, you should analyze just the impacted subset because even a large effect on a small set of users could be diluted until it becomes statistically undetectable. In other words, your analysis sample should include only users eligible for personalization.

One way to think of this is that personalization changes the experience for the 1700 users but not the 483K. If you average the effect for the entire population of users, you are mixing the effect for 1700 with 483K zeros.

Instead, you should randomly split the 1700 personalization-eligible users in half. The first group should be sent the personalized e-mail, and the second the unpersonalized e-mail (or perhaps nothing, depending on what you actually plan to do). The analysis would compare the mean outcomes between the two groups. Since personalization is costly, you may want to make the null that the average/total effect is greater or equal to the average/total cost of personalization rather than zero.

More broadly, this approach is called "triggering" and can improve the sensitivity (statistical power) of the test. The tradeoff is that the estimated effect does not apply to your total user base, just the folks eligible for this particular personalization.

A good reference is Chapter 20 of Kohavi, Tang, and Xu's Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing