# A/B test: measure likelihood of outcomes based on historical events

I am researching if it makes sense to create personalized emails based on past purchase behavior, but I do not know how to design A/B tests. The starting data I have is a list of about 500,000 subscribed email addresses, about 1,700 of which have a common attribute: they've purchased a certain category of product. To do an A/B test, I am considering two options:

1. Let $$A$$ be the 500,000 general list
2. Let $$B$$ be the 1,700 specialized list
3. Let $$E_1$$ be an email template
4. Let $$E_2$$ be a potential second email template

Question:

• Does it make sense to send use one email template, and when sending it, measure and compare engagement of $$A$$ and $$B$$?
• Does it make sense to design two email templates, and for each email template, send half of the total emails sent to $$A$$ and half to $$B$$? And see how each group engages with each email template?

In other words I am consider two strategies:

1. Send 2000 emails using a single template, 1000 to $$A$$, 1000 to $$B$$; compare engagement rates between the groups on the single template; or
2. Create template $$E_1$$ and send 500 to $$A$$ and 500 to $$B$$ using it; create template $$E_2$$ and send 500 to $$A$$ and send 500 to $$B$$. Measure engagement rates for each template and group, and compare.

• Does one make more sense than the other depending on some factor or goal?
• What methods seem appropriate to approximate the significance of any results?

I am considering measuring engagement based simply on how many people click on a link in the email, or perhaps on some event happening on my website such as a product view, a purchase, or a post in the forums.

My experience with statistics is rather minimal. I am in an intro probability theory and statistics class, but we haven't gotten as far as experiment design yet.

Any advice on more helpful details I can include in my question, or alternative approaches to my experiment would be appreciated. Any links to resources on sorting this out would also be helpful, leaning towards an audience with limited statistical background, though having a willingness to learn. (I have some basic probability theory understanding.)

An overall goal is to determine if the effort of gathering this data and transforming it into something usable by our systems is worth it. Perhaps it will take multiple tests to be confident in this approach.

• What is the question your experiment seeks to answer? If successful, would you only send the e-mail to subscribers with the common attribute but not the others? Feb 27 at 19:55
• Identifying common attribute and making it into a usable form for our application takes some effort, so at least part of the goal is to see if this method is worth the effort. I suppose it may take more than one test to determine that generally. A hypothesis is that we have data to make informed decisions here, but I want to know if attributes such as what we're trying here, basing commonality on purchases from a product category makes sense. If it does, we can provide more relevant information to our visitors about what we have to offer. Feb 27 at 21:22
• To elaborate a bit more: I have a lot of subscribers in general, but I am concerned that each email sent is being sent to more than those who would actually be interested. I have news emails, product launch emails, emails about things going on in our website. In each of these categories, there are a lot of subcategories. Many of the subscribers have contributed to the website's public knowledge base or made purchases, so I feel that I can help identify subsets of the subscribers who may be interested in one thing or another, or who may not be interested in certain topics. Feb 27 at 21:45
• My question is about "compared to what"? Is it personalized e-mail vs. sending nothing? Personalized e-mail versus non-personalized e-mail? One type of personalized e-mail versus a second type? Feb 27 at 22:28
• Ah, okay. So right now email is sent that is specific to the website and news, and it's not personalized to subscribers. The intention is to personalize the email. There may be some emails sent that are for a wider audience, but some emails might be sent exclusively to a certain audience that is determined to be more likely interested. So right now, emails are non-personalized, and I am interested in how personalized emails will perform. Furthermore, I am interested in determining if the personalization criteria makes sense, which I think will take trial and error including more experiments Feb 27 at 23:02