Can you combine the results of two A/B tests?

Suppose, a feature of an app is redesigned and A/B tested in Region A with a small population, where x% of this population found the feature to be good, and it improved user engagement on the app.

The same test is then performed in Region B on a larger population, and y% of this population are said to have found the feature to be an improvement, and thus user engagement has increased for those users.

Now, while assessing the A/B test results, should I combine the results of the two regions? If not, how do I go about it? The two regions are largely distinct in terms of the way that users behave on the app in those regions.

• There is a way to do this but it involves some math. I’ll get to this in a moment. Feb 19, 2020 at 0:07
• Sure, I look forward to your approach. Thanks! Feb 19, 2020 at 0:23
• If you ignored your prior knowledge that users behave differently across regions, you could do a Bayesian logistic regression twice, first on region A then on region B and use the posterior that you get from Region A as a prior for region B. However, since you say that you know that users behave differently based on regions, it might be better to use some sort of mixed effects model, although I'm not sure whether it makes sense to estimate a random effect for region when you have only two regions to work with. Feb 19, 2020 at 0:25
• You're using the symbols "A" and "B" to each represent two different things in your question. When you say "A/B testing" the "A" and "B" in that phrase notionally refer to two things being compared in a test within each region. If you also use to to label the regions themselves, this is going to lead to ambiguity. If you must use the phrase "A/B testing" $^\dagger$, avoid using "A" and "B" for anything else but the two things being compared in that test. $\: ... \: \dagger$ (perhaps a poor choice when more specific alternatives already existed at its coining, but we're stuck with it now), Feb 19, 2020 at 4:27

If you had more than two groups, you could do what is called a meta-analysis. That is where you take the observed effects across different populations and pool them to get a population effect (I'm being very fast and loose here).

Because you have two experiments, there is no good justification for doing a meta analysis and so I think your best bet is to simply fit a mixed model (be it Bayesian or otherwise). Here is a small example:

In the two populations, say I made 100 and 120 clicks on 6000 and 5500 impressions respectively. This results in a 1.7% and 2.2% click through ratio. These estimates are variable because they come from different populations. The goal is to estimate the click through ratio for the ad marginalized over populations.

Here is the code to do so:

library(tidyverse)
library(rstanarm)
library(lme4)

inv.logit = function(x) 1/(1+exp(-x))

d = tribble(
~'click', ~'impression', ~'population',
100, 6000, 'A',
120, 5500, 'B'
)

model = glmer(cbind(click, impression-click) ~ 1 + (1|population),
data = d,
family = binomial())

model %>% summary


The coefficient for the intercept of this model is our estimate for the click through rate (on the log odds scale). To get the probability, we can do inv.logit(fixef(model)) which returns a probability of 1.9%.

An even easier way would be to do a weighted average of the CTR. Weight the CTR by their variance and then average them.

d$$p = d$$click/d$impression d$$var = d$$p*(1-d$$p)/d$$impression d$$w = d$$var/sum(d$var)

est = d$$p %*% d$$w

est


Similar number, no confidence intervals this way (at least, not easily).

As an aside, I don't think doing this is a great idea. If users are distinct between regions, then including more regions into this analysis will yield a better estimate of the population level CTR. Its obviously your decision, but this doesn't sound robust to me.