Can you combine the results of two A/B tests?

Question

Suppose, a feature of an app is redesigned and A/B tested in Region A with a small population, where x% of this population found the feature to be good, and it improved user engagement on the app.

The same test is then performed in Region B on a larger population, and y% of this population are said to have found the feature to be an improvement, and thus user engagement has increased for those users.

Now, while assessing the A/B test results, should I combine the results of the two regions? If not, how do I go about it? The two regions are largely distinct in terms of the way that users behave on the app in those regions.

Answer 1

If you had more than two groups, you could do what is called a meta-analysis. That is where you take the observed effects across different populations and pool them to get a population effect (I'm being very fast and loose here).

Because you have two experiments, there is no good justification for doing a meta analysis and so I think your best bet is to simply fit a mixed model (be it Bayesian or otherwise). Here is a small example:

In the two populations, say I made 100 and 120 clicks on 6000 and 5500 impressions respectively. This results in a 1.7% and 2.2% click through ratio. These estimates are variable because they come from different populations. The goal is to estimate the click through ratio for the ad marginalized over populations.

Here is the code to do so:

library(tidyverse)
library(rstanarm)
library(lme4)

inv.logit = function(x) 1/(1+exp(-x))

d = tribble(
  ~'click', ~'impression', ~'population',
  100, 6000, 'A',
  120, 5500, 'B'
)

model = glmer(cbind(click, impression-click) ~ 1 + (1|population),
                   data = d,
                   family = binomial())

model %>% summary

The coefficient for the intercept of this model is our estimate for the click through rate (on the log odds scale). To get the probability, we can do inv.logit(fixef(model)) which returns a probability of 1.9%.

An even easier way would be to do a weighted average of the CTR. Weight the CTR by their variance and then average them.

d$p = d$click/d$impression

d$var = d$p*(1-d$p)/d$impression

d$w = d$var/sum(d$var)

est = d$p %*% d$w

est

Similar number, no confidence intervals this way (at least, not easily).

As an aside, I don't think doing this is a great idea. If users are distinct between regions, then including more regions into this analysis will yield a better estimate of the population level CTR. Its obviously your decision, but this doesn't sound robust to me.