You can use mixed effects models here. For
CONVERSION, this would be a generalised linear mixed model (glmm), which will provide a (log) odds ratio, z test and p-value.
If the idea of odds and odds ratios is new to you, a small example should help. Suppose we observe these data, ignoring the issue of dependence within users:
It is fairly obvious from inspection that group B has a higher conversion rate than group A. Specifically, 5/7 vs 2/6.
In R, with these data in a dataframe called
dt, we can easily produce the following contingency table:
> xtabs(~ GROUP + CONVERSION, data = dt)
GROUP 0 1
A 4 2
B 2 5
Now we can more easily talk about odds. The Odds of an event is probability of the event happening divided by the probability of the same event not happening. Looking at the contingency table, we see that for group A, the odds of conversion are 2/4 = 0.5, while for group B the odds of conversion are 5/2 = 2.5. We say that the "odds ratio" is 2.5/0.5 = 5. In other words, the odds of conversion for group B are 5 times higher than the odds of conversion for group A. It should be obvious that if the probabilities were equal in both groups, the odds ratio would be 1 so when the odds ratio is above or below 1, we have higher odds in one group than the other.
With the toy dataset
dt, we can now fit the following model:
> glm1 <- glm(CONVERSION ~ GROUP, data = dt, family = binomial)
where we specify the binomial family of distributions because the outcome is binary. This fits a generalised linear model (glm) using the default logit link, with the following output:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -0.6931 0.8660 -0.800 0.423
GROUPB 1.6094 1.2042 1.337 0.181
Note that by default the estimates are on the log scale so we can simply exponentiate:
which tells us that the odds ratio is 5, as we computed above by hand. Note also that the output from
summary also provides a z statistic along with it's p-value, which in this case is fairly high due to the low sample size. This tests the null hypothesis that the estimate is zero (on the log scale, which corresponds to 1 on the odds ratio scale, that being equal probability of conversion)
In your case, we need to account for dependence within users, so we fit random intercepts for
USER, using a function from a mixed effects model package, such as
GLMMAdaptive::mixed_model and the formula would be:
CONVERSION ~ GROUP + (1|USER), family = binomial(link = logit)
Moving on to
SPEND, this is a numeric variable, so, at least in the first instance, we can fit a regular linear mixed model with
SPEND ~ GROUP + (1|USER)
Here there is no need to specify a distribution family, and as before you would focus on the estimate for
GROUP which would be interpreted in the same way as a standard linear regression model. That is, the estimate for
GROUP would be the difference in
SPEND for group B, compared to group A. Again, the software will provide a test of the null hypothesis that this estimate is equal to zero.