How to use multivariate analysis to analyse two simultaneous A/B tests

I am analysing a simple email campaign. We are testing two different subject lines, and two different email bodies. Each combination of subject-body (four total combinations) are sent to equal proportions of our email base, across equally-distribution geographical locations. I would like to know the influence that each subject line, and each email body, has on the response rate.

After sending the emails, we will have the following table:

---------------------------------
Subject A | Subject B |
Email A |   x_aa    |   x_ba    |
Email B |   x_ab    |   x_bb    |
---------------------------------


Where $$x_{a,a}...x_{b,b}$$ is the response rate for each subject-body combination.

I've read the following question on this site:

And I tried googling, but I can't find many resources about manually performing multivariate analysis, or even what the actual statistical methods are. I came across this paper, "Discrete Multivariate Analysis: Theory and Practice" but it's way over my head.

How can I calculate the relative influence of a given subject line, and email body, on the total response rate? (And does this analysis require a larger sample size than traditional AB testing?)

Also, if you know of any high-quality and accessible resources to learn this/multivariate analysis please share!

Not sure what you mean by "manually". Maybe pen and paper? I think most tests that could be done pen and paper would likely tell you if a difference exists, but perhaps not what the best combination of exposures is. For that, I think you should use a statistical model. Response rates in my experience are quite low, so I don't think a linear probability model (i.e. using linear regression on proportion data) is a good idea. But, the steps would remain the same for the linear probability model in the case the response rates are relatively large.

Working with logistic regression, you want to model

$$\operatorname{logit}(\mbox{response probablity}) = \beta_0 + \beta_1\mbox{variant subject} + \beta_2 \mbox{variant email} + \beta_3 \mbox{variant subject} \times \mbox{variant emial}$$

Here, $$\mbox{variant subject}$$ and $$\mbox{variant email}$$ are binary indicators (i.e. 1s and 0s). You can then produce probability estimates for each exposure.

Logistic regression will give you the probability an individual responds to/clicks through/opens (whatever the outcome is. In the model, I call it "responding to") the email. Multiply this by the number of people you intend to send the emails to and this gives you the estimated response rate. Because you have an interaction, you are going to need more data than you might expect. I think we can reserve discussion for that later.

Here is an example in R

      y email subject     n
<int> <int>   <int> <int>
1     0     0       0  2371
2     0     0       1  2205
3     0     1       0  2322
4     0     1       1  2167
5     1     0       0   157
6     1     0       1   268
7     1     1       0   170
8     1     1       1   340


Here, y is the outcome (1 for a response, 0 for no response), email and subject are indicators as I've specified, and n is the number of people for which the outcome was observed with those covariates. The model is fit using

model = glm(y ~ email*subject, data = ab_test, family = binomial(), weights = n)


You can look at summary(model) to look at coefficients if you like. You can read up on logistic regression to understand what coefficients mean.

Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept)   -2.71482    0.08241 -32.943  < 2e-16 ***
email          0.10044    0.11447   0.877    0.380
subject        0.60733    0.10477   5.797 6.75e-09 ***
email:subject  0.15491    0.14385   1.077    0.282


Looks like the subject exposures was significant, but the email exposure and the interaction were not. Because you have an interaction in the model, the coefficient table doesn't tell the whole story. I would do a likelihood ratio test to see if the email had any effect whatsoever.

Analysis of Deviance Table

Model 1: y ~ email * subject
Model 2: y ~ subject
Resid. Df Resid. Dev Df Deviance Pr(>Chi)
1         4     6104.8
2         6     6114.3 -2  -9.4149 0.009028 **
---



We reject the null for this test: Having email and the interaction in the model improves the fit over just having subject alone.

I find plotting the probabilities and the confidence intervals easier to interpret

grid = crossing(email = c(0,1), subject = c(0,1))
predictions = predict(model, newdata = grid, se.fit = T)

bind_cols(grid, predictions) %>%
mutate(groups = factor(c('Control','Subject Only','Email Only','Both'), ordered = T, levels = c('Control','Subject Only','Email Only','Both'))) %>%
mutate(prob = plogis(fit),
ci_u = plogis(fit + 2*se.fit),
ci_l = plogis(fit - 2*se.fit)) %>%
ggplot(aes(groups, prob, ymin = ci_l, ymax = ci_u))+
geom_pointrange()+
labs(x = 'Exposure',y = 'Probability')+
scale_y_continuous(labels = scales::percent)


From this plot, you can see that the subject exposure increases the response rate by quite a bit. The email only exposure increases the response rate a little bit, but the uncertainty in the email only estimate makes it seem like it isn't much different from the control group. But using both email and subject exposures creates a synergistic effect, almost doubling the response rate as compared to email alone.