2
$\begingroup$

I'd like to run an A/B experiment where my outcome measures are conversion rate, and mean spend. I'd like to use two proportion Z-test and z-test of two population means, however many users in the study have multiple observations, so therefore it is not iid. I could obviously limit it to 1 record per user, but I'd prefer not to.

Users were simple random sampled into 2 groups (A and B). The dataframe is structured as so, where each row is an observation (about 30000 observations and about 5000 users):

GROUP | USER | CONVERSION | SPEND ($)
  A     1001       1         250
  A     1050       0         320
  B     1231       1         215
  A     1001       1         245
  A     1001       0         275

Are there any statistical techniques to get around this? I've heard of using mixed effects models for similar problems, but I don't know if that's applicable to these two hypothesis tests and how I would frame it

Learning resources also appreciated!

$\endgroup$
3
  • 1
    $\begingroup$ Can you add some details about exactly how the test was conducted and about all the variables that were collected/measured/observed ? $\endgroup$ Dec 13, 2019 at 10:29
  • $\begingroup$ just updated the question to include an example dataframe. In reality, there are >5000 users and >30000 observations $\endgroup$
    – J Doe
    Dec 13, 2019 at 22:09
  • $\begingroup$ OK. So, I'm assuming this is to test 2 different version of a website... Just to be clear, the difference between the 2 groups is that one used website A (group A), and one used website B (group B) ? Your interest is in whether those in group A were more likely to have conversion=1, than conversion=0 (or the other way around), and also whether being in group A vs group B was associated with a larger (or smaller) spend, and you want to account for dependence within users while performing these tests ? $\endgroup$ Dec 14, 2019 at 11:36

1 Answer 1

2
$\begingroup$

You can use mixed effects models here. For CONVERSION, this would be a generalised linear mixed model (glmm), which will provide a (log) odds ratio, z test and p-value.

If the idea of odds and odds ratios is new to you, a small example should help. Suppose we observe these data, ignoring the issue of dependence within users:

GROUP   CONVERSION
A        0
A        0
A        0
A        0
A        1
A        1
B        1
B        1
B        1
B        0
B        0
B        1
B        1

It is fairly obvious from inspection that group B has a higher conversion rate than group A. Specifically, 5/7 vs 2/6.

In R, with these data in a dataframe called dt, we can easily produce the following contingency table:

> xtabs(~ GROUP + CONVERSION, data = dt)  

     CONVERSION
GROUP   0     1
    A   4     2
    B   2     5

Now we can more easily talk about odds. The Odds of an event is probability of the event happening divided by the probability of the same event not happening. Looking at the contingency table, we see that for group A, the odds of conversion are 2/4 = 0.5, while for group B the odds of conversion are 5/2 = 2.5. We say that the "odds ratio" is 2.5/0.5 = 5. In other words, the odds of conversion for group B are 5 times higher than the odds of conversion for group A. It should be obvious that if the probabilities were equal in both groups, the odds ratio would be 1 so when the odds ratio is above or below 1, we have higher odds in one group than the other.

With the toy dataset dt, we can now fit the following model:

> glm1 <- glm(CONVERSION ~ GROUP, data = dt, family = binomial)

where we specify the binomial family of distributions because the outcome is binary. This fits a generalised linear model (glm) using the default logit link, with the following output:

> summary(glm1)

Coefficients:
            Estimate Std. Error z value Pr(>|z|)
(Intercept)  -0.6931     0.8660  -0.800    0.423
GROUPB        1.6094     1.2042   1.337    0.181

Note that by default the estimates are on the log scale so we can simply exponentiate:

> exp(coef(glm1)[2])
GroupB 
 5

which tells us that the odds ratio is 5, as we computed above by hand. Note also that the output from summary also provides a z statistic along with it's p-value, which in this case is fairly high due to the low sample size. This tests the null hypothesis that the estimate is zero (on the log scale, which corresponds to 1 on the odds ratio scale, that being equal probability of conversion)

In your case, we need to account for dependence within users, so we fit random intercepts for USER, using a function from a mixed effects model package, such as lme4::glmer or GLMMAdaptive::mixed_model and the formula would be:

CONVERSION ~ GROUP + (1|USER), family = binomial(link = logit)

Moving on to SPEND, this is a numeric variable, so, at least in the first instance, we can fit a regular linear mixed model with lme4::lmer or GLMMAdaptive::mixed_model:

SPEND ~ GROUP + (1|USER)

Here there is no need to specify a distribution family, and as before you would focus on the estimate for GROUP which would be interpreted in the same way as a standard linear regression model. That is, the estimate for GROUP would be the difference in SPEND for group B, compared to group A. Again, the software will provide a test of the null hypothesis that this estimate is equal to zero.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.