Not sure what you mean by "manually". Maybe pen and paper? I think most tests that could be done pen and paper would likely tell you if a difference exists, but perhaps not what the best combination of exposures is. For that, I think you should use a statistical model. Response rates in my experience are quite low, so I don't think a linear probability model (i.e. using linear regression on proportion data) is a good idea. But, the steps would remain the same for the linear probability model in the case the response rates are relatively large.
Working with logistic regression, you want to model
$$ \operatorname{logit}(\mbox{response probablity}) = \beta_0 + \beta_1\mbox{variant subject} + \beta_2 \mbox{variant email} + \beta_3 \mbox{variant subject} \times \mbox{variant emial}$$
Here, $\mbox{variant subject}$ and $\mbox{variant email}$ are binary indicators (i.e. 1s and 0s). You can then produce probability estimates for each exposure.
Logistic regression will give you the probability an individual responds to/clicks through/opens (whatever the outcome is. In the model, I call it "responding to") the email. Multiply this by the number of people you intend to send the emails to and this gives you the estimated response rate. Because you have an interaction, you are going to need more data than you might expect. I think we can reserve discussion for that later.
Here is an example in R
y email subject n
<int> <int> <int> <int>
1 0 0 0 2371
2 0 0 1 2205
3 0 1 0 2322
4 0 1 1 2167
5 1 0 0 157
6 1 0 1 268
7 1 1 0 170
8 1 1 1 340
Here, y is the outcome (1 for a response, 0 for no response), email and subject are indicators as I've specified, and n is the number of people for which the outcome was observed with those covariates. The model is fit using
model = glm(y ~ email*subject, data = ab_test, family = binomial(), weights = n)
You can look at summary(model) to look at coefficients if you like. You can read up on logistic regression to understand what coefficients mean.
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -2.71482 0.08241 -32.943 < 2e-16 ***
email 0.10044 0.11447 0.877 0.380
subject 0.60733 0.10477 5.797 6.75e-09 ***
email:subject 0.15491 0.14385 1.077 0.282
Looks like the subject exposures was significant, but the email exposure and the interaction were not. Because you have an interaction in the model, the coefficient table doesn't tell the whole story. I would do a likelihood ratio test to see if the email had any effect whatsoever.
Analysis of Deviance Table
Model 1: y ~ email * subject
Model 2: y ~ subject
Resid. Df Resid. Dev Df Deviance Pr(>Chi)
1 4 6104.8
2 6 6114.3 -2 -9.4149 0.009028 **
---
We reject the null for this test: Having email and the interaction in the model improves the fit over just having subject alone.
I find plotting the probabilities and the confidence intervals easier to interpret
grid = crossing(email = c(0,1), subject = c(0,1))
predictions = predict(model, newdata = grid, se.fit = T)
bind_cols(grid, predictions) %>%
mutate(groups = factor(c('Control','Subject Only','Email Only','Both'), ordered = T, levels = c('Control','Subject Only','Email Only','Both'))) %>%
mutate(prob = plogis(fit),
ci_u = plogis(fit + 2*se.fit),
ci_l = plogis(fit - 2*se.fit)) %>%
ggplot(aes(groups, prob, ymin = ci_l, ymax = ci_u))+
geom_pointrange()+
labs(x = 'Exposure',y = 'Probability')+
scale_y_continuous(labels = scales::percent)
From this plot, you can see that the subject exposure increases the response rate by quite a bit. The email only exposure increases the response rate a little bit, but the uncertainty in the email only estimate makes it seem like it isn't much different from the control group. But using both email and subject exposures creates a synergistic effect, almost doubling the response rate as compared to email alone.
