How to compare conversion rates in an AB test?

Question

Please help me decide how to proceed in the following situation:

• Suppose we have an AB test with 2 groups.
• The target metric is the conversion from one web page to another.
• Suppose also, that whenever a user enters the website, we are able to identify his ID correctly.
• Every user can go through the "flow" arbitrary number of times (i.e. he can do the target action 3 out of 7 times).

So here is my question: Should I:

1. Calculate the overall conversion rate for both groups and compare it via a simple z-test?

2. Calculate the overall conversion rate for each user and then compare the average values across groups via a t-test?

Intuitively, users with a larger number of actions are given more weight in the first option, while in the latter everyone carries equal weight.

Finally, which option do I choose?

Answer 1

I am going to cook up an admittedly extreme answer to demonstrate my point. Let's say you run an AB test on your platform to examine conversion. Whatever you are measuring can be accessed several times, and so you see the same customer pop up in your data more than once.

We randomize 10 users to versions A or B, and then perform a hypothesis test. Here is some data I simulated

# A tibble: 2 x 3
  group     z     n
  <chr> <int> <int>
1 A        23    50
2 B       159   230

Here, the sum of the $n$ column is larger than 10 because I see experimental units more than once (as in your first approach). If I did a test of proportions on this data, then I would reject the null with a p value of about 0.003. Wow, there is less than a 3 in 1000 chance we get this result assuming there is no difference. We should ship this change!

Or should we? Let's see how many times each subject was observed.

  users     n
   <fct> <int>
 1 1        10
 2 2        10
 3 3        10
 4 4        10
 5 5        10
 6 6        10
 7 7        10
 8 8        10
 9 9       100
10 10      100

We saw 8 of the 10 experimental units a total of 10 times, but we saw the last two units 100 times! If these users were more likely to convert anyway regardless of experimental arm, then our results are biased! And that is exactly what happened.

# A tibble: 10 x 4
   group users     z     n
   <chr> <fct> <int> <int>
 1 A     1         4    10
 2 A     2         4    10
 3 A     3         5    10
 4 A     4         4    10
 5 A     5         6    10
 6 B     6         7    10
 7 B     7         6    10
 8 B     8         3    10
 9 B     9        74   100
10 B     10       69   100

These users were just more likely to convert even before the experiment. This happens in real life. If you buy a lot of stuff on Amazon, for example, you probably are not going to be affected by experimental arm, which can lead to stuff like this happening.

How can we get around this? Previously, the more we saw an experimental unit, the more weight they had. To weight everyone accordingly we can estimate each person's probability of conversion and then do a t test. When we do that, we correctly fail to reject the null (when I generated this data, I made sure there was no difference between groups but that experimental units had their own unique probability of conversion).

Answer 2

Let's refine the design of the experiment: in a typical ab test, the experiment is randomized by user, this means that the group to which the user belongs does not change during the overall experiment. Randomize by visit is possible, but not commonly used: this means giving the same user inconsistent treatment, and may violate the Stable Unit Treatment Value Assumption (SUTVA). The following discussion will be based on the first design.

Here is a toy data:

  user_cnt <- 100

  ## each user can go through the "flow" many number of times
  visit_oer_user <- rpois(user_cnt, 10) 
  
  ## each each have unequal probability to convert
  user_conversion_rate <- runif(user_cnt, 0.0, 0.1) 
  visit_cnt <- sum(visit_oer_user)
  
  ## multiple visit of a user have equal probability to convert
  visit_conversion_rate <- rep(user_conversion_rate, visit_oer_user) 
  visit_id <- seq(1, visit_cnt)
  user_id <- rep(1:user_cnt, visit_oer_user)
  
  ## randomized by user, not visit
  user_group <- sample(0:1, user_cnt, replace = TRUE)
  visit_group <- rep(user_group, visit_oer_user)
  
  y <- rbinom(visit_cnt, 1, visit_conversion_rate)
  
  df <- tibble(
    visit_id,
    user_id,
    trt = visit_group,
    y
  )
  df    

Both options are common used but represent different meanings: conversion-by-visit vs conversion-by-user. If you choose the first option(conversion-by-visit), you need to use more nuanced analyses methods such as bootstrap, delta method or cluster robust standard errors, for violates the assumption of iid. An example in R would like this:

library(sandwich)
library(lmtest)

ols_mod <- lm(y ~ trt, df)
cluster_mod <- coeftest(ols_mod, vcov = vcovCL, cluster = ~user_id)
cluster_mod

t test of coefficients:

             Estimate Std. Error t value Pr(>|t|)   
(Intercept)  0.097561   0.034858  2.7988 0.005631 **
trt         -0.047561   0.039602 -1.2010 0.231182   
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

by the second option(conversion-by-user) is easier: aggregate data to user level, apply t test or linear regression.

df_user <- df %>% 
  group_by(user_id, trt) %>% 
  summarise(
    y = max(y),
    .groups = 'drop'
  )
ols <- lm(y ~ trt, df)
summary(ols)

Call:
lm(formula = y ~ trt, data = df)

Residuals:
     Min       1Q   Median       3Q      Max 
-0.09756 -0.09756 -0.05000 -0.05000  0.95000 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  0.09756    0.02807   3.476 0.000624 ***
trt         -0.04756    0.03642  -1.306 0.193031    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.2542 on 200 degrees of freedom
Multiple R-squared:  0.008457,  Adjusted R-squared:  0.003499 
F-statistic: 1.706 on 1 and 200 DF,  p-value: 0.193

references:

Deng, A., Lu, J., & Qin, W. (2021). The equivalence of the Delta method and the cluster-robust variance estimator for the analysis of clustered randomized experiments. arXiv preprint arXiv:2105.14705.

Deng, A., Lu, J., & Litz, J. (2017, February). Trustworthy analysis of online a/b tests: Pitfalls, challenges and solutions. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining (pp. 641-649).