I am researching if it makes sense to create personalized emails based on past purchase behavior, but I do not know how to design A/B tests. The starting data I have is a list of about 500,000 subscribed email addresses, about 1,700 of which have a common attribute: they've purchased a certain category of product. To do an A/B test, I am considering two options:

  1. Let $A$ be the 500,000 general list
  2. Let $B$ be the 1,700 specialized list
  3. Let $E_1$ be an email template
  4. Let $E_2$ be a potential second email template


  • Does it make sense to send use one email template, and when sending it, measure and compare engagement of $A$ and $B$?
  • Does it make sense to design two email templates, and for each email template, send half of the total emails sent to $A$ and half to $B$? And see how each group engages with each email template?

In other words I am consider two strategies:

  1. Send 2000 emails using a single template, 1000 to $A$, 1000 to $B$; compare engagement rates between the groups on the single template; or
  2. Create template $E_1$ and send 500 to $A$ and 500 to $B$ using it; create template $E_2$ and send 500 to $A$ and send 500 to $B$. Measure engagement rates for each template and group, and compare.

To follow up

  • Are there advantages or disadvantages to either approach?
  • Does one make more sense than the other depending on some factor or goal?
  • What methods seem appropriate to approximate the significance of any results?

I am considering measuring engagement based simply on how many people click on a link in the email, or perhaps on some event happening on my website such as a product view, a purchase, or a post in the forums.

My experience with statistics is rather minimal. I am in an intro probability theory and statistics class, but we haven't gotten as far as experiment design yet.

Any advice on more helpful details I can include in my question, or alternative approaches to my experiment would be appreciated. Any links to resources on sorting this out would also be helpful, leaning towards an audience with limited statistical background, though having a willingness to learn. (I have some basic probability theory understanding.)

An overall goal is to determine if the effort of gathering this data and transforming it into something usable by our systems is worth it. Perhaps it will take multiple tests to be confident in this approach.

  • 1
    $\begingroup$ What is the question your experiment seeks to answer? If successful, would you only send the e-mail to subscribers with the common attribute but not the others? $\endgroup$
    – dimitriy
    Commented Feb 27, 2023 at 19:55
  • $\begingroup$ Identifying common attribute and making it into a usable form for our application takes some effort, so at least part of the goal is to see if this method is worth the effort. I suppose it may take more than one test to determine that generally. A hypothesis is that we have data to make informed decisions here, but I want to know if attributes such as what we're trying here, basing commonality on purchases from a product category makes sense. If it does, we can provide more relevant information to our visitors about what we have to offer. $\endgroup$
    – Addison
    Commented Feb 27, 2023 at 21:22
  • $\begingroup$ To elaborate a bit more: I have a lot of subscribers in general, but I am concerned that each email sent is being sent to more than those who would actually be interested. I have news emails, product launch emails, emails about things going on in our website. In each of these categories, there are a lot of subcategories. Many of the subscribers have contributed to the website's public knowledge base or made purchases, so I feel that I can help identify subsets of the subscribers who may be interested in one thing or another, or who may not be interested in certain topics. $\endgroup$
    – Addison
    Commented Feb 27, 2023 at 21:45
  • 1
    $\begingroup$ My question is about "compared to what"? Is it personalized e-mail vs. sending nothing? Personalized e-mail versus non-personalized e-mail? One type of personalized e-mail versus a second type? $\endgroup$
    – dimitriy
    Commented Feb 27, 2023 at 22:28
  • $\begingroup$ Ah, okay. So right now email is sent that is specific to the website and news, and it's not personalized to subscribers. The intention is to personalize the email. There may be some emails sent that are for a wider audience, but some emails might be sent exclusively to a certain audience that is determined to be more likely interested. So right now, emails are non-personalized, and I am interested in how personalized emails will perform. Furthermore, I am interested in determining if the personalization criteria makes sense, which I think will take trial and error including more experiments $\endgroup$
    – Addison
    Commented Feb 27, 2023 at 23:02

1 Answer 1


If the treatment impacts only a tiny fraction of users, you should analyze just the impacted subset because even a large effect on a small set of users could be diluted until it becomes statistically undetectable. In other words, your analysis sample should include only users eligible for personalization.

One way to think of this is that personalization changes the experience for the 1700 users but not the 483K. If you average the effect for the entire population of users, you are mixing the effect for 1700 with 483K zeros.

Instead, you should randomly split the 1700 personalization-eligible users in half. The first group should be sent the personalized e-mail, and the second the unpersonalized e-mail (or perhaps nothing, depending on what you actually plan to do). The analysis would compare the mean outcomes between the two groups. Since personalization is costly, you may want to make the null that the average/total effect is greater or equal to the average/total cost of personalization rather than zero.

More broadly, this approach is called "triggering" and can improve the sensitivity (statistical power) of the test. The tradeoff is that the estimated effect does not apply to your total user base, just the folks eligible for this particular personalization.

A good reference is Chapter 20 of Kohavi, Tang, and Xu's Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.