I want to construct an ML algorithm that replace the classic A/B testing, actually a predictive model that replace the A/B testing with very high accuracy, I have about 10.000 rows of emails subject lines with their open rate. My goal is to build an algorithm that takes 2 subject lines of emails as input and returns the best subject line.

First of all, does this data contains the proper information for the project?

I did a quick research on the internet and I found that t-test or Chi-Squared tests are used for A/B testing, but unlikely that can't work on my project. Then I found the Bayesian models but I could not understand them...

I think that to use Bayesian models you have to perform a A/B test for example for the color of the button, collect the data and then use that model to find the proper answer.

Any ideas or resources about that?

Does anyone know how to approach this problem?

  • $\begingroup$ Testing and predictive modeling are different things -- what precisely do you want to achieve? $\endgroup$ – Tim May 12 '17 at 10:03

I am reading your problem as follows:

You have data with email subject lines and their open rates. You want to know how subject lines predict open rates. You have no theory about this.

You mention A/B testing, which is a hypothesis-driven approach, in which you test a method/tool/website/advertisement/... in two variants and evaluate which one - according to a given criterion (accuracy, outreach, revenue,...) performs better. In your example, you could posit that emails containing the word "IMPORTANT" increase the open rate in contrast to those that do not. You would probably evaluate the evidence for A over B using a null hypothesis test (but of course, there are alternatives).

When employing a predictive model, you are switching to an exploratory, data-driven road, in which no such A/B-condition is set a priori but you try to learn from the data what regularities in your data best predict your outcome. In other words, you want to find a function 'f' that maps subject lines to open rate. This would enable you to run a pair of email subjects X and Y through your function and compare which is larger, f(x) or f(y).

A comparison of a statistical hypothesis testing perspective on group differences VS classifiers as means for performing group comparisons was recently given by Kim and von Oertzen. You find the full article as: Kim, B. & Oertzen, T.. Behav Res (2017). doi:10.3758/s13428-017-0880-z

For a start, you probably should look at algorithms for spam detection as they contain most of the functionality you need, e.g., tokenizing an email or subject line, preprocessing of words, naive Bayes classifiers. However, note that you are really dealing with a regression problem here. There are many different ways to tackle this type of regression problem and I recommend to start with basic reading on machine learning and/or data science classes.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.