I am researching if it makes sense to create personalized emails based on past purchase behavior, but I do not know how to design A/B tests. The starting data I have is a list of about 500,000 subscribed email addresses, about 1,700 of which have a common attribute: they've purchased a certain category of product. To do an A/B test, I am considering two options:
- Let $A$ be the 500,000 general list
- Let $B$ be the 1,700 specialized list
- Let $E_1$ be an email template
- Let $E_2$ be a potential second email template
- Does it make sense to send use one email template, and when sending it, measure and compare engagement of $A$ and $B$?
- Does it make sense to design two email templates, and for each email template, send half of the total emails sent to $A$ and half to $B$? And see how each group engages with each email template?
In other words I am consider two strategies:
- Send 2000 emails using a single template, 1000 to $A$, 1000 to $B$; compare engagement rates between the groups on the single template; or
- Create template $E_1$ and send 500 to $A$ and 500 to $B$ using it; create template $E_2$ and send 500 to $A$ and send 500 to $B$. Measure engagement rates for each template and group, and compare.
To follow up
- Are there advantages or disadvantages to either approach?
- Does one make more sense than the other depending on some factor or goal?
- What methods seem appropriate to approximate the significance of any results?
I am considering measuring engagement based simply on how many people click on a link in the email, or perhaps on some event happening on my website such as a product view, a purchase, or a post in the forums.
My experience with statistics is rather minimal. I am in an intro probability theory and statistics class, but we haven't gotten as far as experiment design yet.
Any advice on more helpful details I can include in my question, or alternative approaches to my experiment would be appreciated. Any links to resources on sorting this out would also be helpful, leaning towards an audience with limited statistical background, though having a willingness to learn. (I have some basic probability theory understanding.)
An overall goal is to determine if the effort of gathering this data and transforming it into something usable by our systems is worth it. Perhaps it will take multiple tests to be confident in this approach.