What test/model would I use for A/B testing given 3 groups?

Question

Problem: Analyze an experiment that was developed for the iOS app, called “ios_referral_experiment”, that tests adding new links to the referral invite page. The experiment has 3 groups: the “control” group has no new link; the “tab_only” group has a new navigation tab that links to the page; and the “tab_settings” group has both a new navigation tab and a new link in the settings. What are the results from this experiment and which group would you recommend based on the results?

Data Parameters:

This table includes click and view events in the UI.

● user_id: unique id for the user

● exposed_time: time when the user was exposed to the experiment

● split_test_group: the split test group the user was assigned to events

● event_time: time when the event occurred

● event_type: the type of event that happened

○ “referral_page_view”: user views the page for sending referral invites

○ “referrer_page_invite_action”: users clicks the button to send a referral invite

● event_type_button: the type of button the user clicks for the event referrals

                             user_id           split_test_group exposed_time                             event_type     platform       event_type_button       Date                Time          
 b861f69cb669766cc0d52bc2279332c0:   414   control     :27274   Length:142123      referrer_page_invite_action:  8896   iOS:135067              :133227   Min.   :2017-06-02   Length:142123     
 5ac762996eb6e8932fd2140e77a7a870:   376   tab_only    :67381   Class :character   referrer_page_viewed       :133227   Web:  7056   Link       :  1816   1st Qu.:2017-07-21   Class :character  
 00c26d8255da64576a9e3c5a2c1271eb:   298   tab_settings:47468   Mode  :character                                                     SMS        :  1727   Median :2017-08-04   Mode  :character  
 2d4fdc1606ad722a93a98882f9ccf331:   236                                                                                             link       :  1226   Mean   :2017-08-02                     
 4296772f2a0ce768d25573863c82968c:   236                                                                                             email      :  1118   3rd Qu.:2017-08-17                     
 fe7a74a142b5774375ba64184d67a8e3:   212                                                                                             ShareDialog:  1062   Max.   :2017-09-01                     
 (Other)                         :140351                                                                                             (Other)    :  1947           

I would like to calculate based on each 'split_test_group' - which had the greatest influence on the number of 'referrer_page_invite_action' vs 'referrer_page_viewed'.

Update

New data table based on count per group

split_test_group   |   event_type                  |     count

Control                referrer_page_invite_action       1892
Control                referrer_page_viewed              25382
tab_only               referrer_page_invite_action       4009
tab_only               referrer_page_viewed              63372
tab_settings           referrer_page_invite_action       2995
tab_settings           referrer_page_viewed              44473
    
```

Answer 1

Doing a joint optimization for two metrics is an interesting problem, but unfortunately a bit out my expertise. However, making a decision based on a single metric is straight forward enough. Here is a simulated example of how this might be done using a Bayesian lense.

Let's first simulate some data

library(tidyverse)

set.seed(0)
N = 10000
groups = sample(LETTERS[1:3], size = N, replace = T)

# Generate fake data
X = model.matrix(~groups)
true_beta = c(qlogis(0.1), qlogis(0.12) - qlogis(0.1), qlogis(0.09) - qlogis(0.1))
true_p = plogis(X %*% true_beta)

# The observations
y = rbinom(N, 1, true_p)
d = tibble(groups, y)

In our fake experiment, we have allocated approximately 3300 people to one of 3 groups and measured a binary outcome. Here is a summary of the data.

  groups     N     y
  <chr>  <int> <int>
1 A       3385   359
2 B       3311   415
3 C       3304   297

To begin with, we need a loss function, essentially a function which tells us what I would stand to lose were I to make a decision. Each action, including the do nothing scenario (where we just keep the control group as the main exposure) is a decision, so we need to know what we would lose if that were the wrong decision.

We really don't want to hurt the rate/metric at which the binary outcome happens (whatever that rate/metric is). One loss function we might find interesting is the following

$$ \mathcal{L}(x, p_a, p_b, p_c) = \begin{cases} \operatorname{max}(\operatorname{max}(p_b, p_c) - p_a, 0) & \quad x=a \\ \operatorname{max}(\operatorname{max}(p_a, p_c) - p_b, 0) & \quad x=b\\ \operatorname{max}(\operatorname{max}(p_a, p_b) - p_c, 0) & \quad x=c \end{cases}$$

Here, $x$ is the variant we choose to implement, and $p_i$ is the rate for group $i$. The function $\mathcal{L}$ tells us how much we would stand to hurt our metric if we chose the wrong variant. So for example, let's say we choose variant $a$. In the case where $a$ truly is the superior variant ($p_a>p_b$ and $p_a>p_c$) then $\max{\max{p_b, p_c} - p_a, 0} = 0$ and so we lose out on a total 0 change to the metric. But suppose that $p_b>p_a$, then choosing $a$ means we lose out on $p_b-p_a$. Had we chosen $b$, our metric would have been higher, but since we didn't we lose out on that increase.

The thing is that $p_a$, $p_b$, and $p_c$ are all unknown, so we have to estimate them from our data. We can do that fairly easily using R and some tools for logistic regression (I am using Bayesian methods because it makes the decision process very easy as we will see).

Using the data created above, we can estimate a logistic regression with

# Bayesian models library
library(brms)
library(tidybayes)

# Modify data so we have success/trials as our coliumns
modified_d = group_by(d, groups) %>% summarise(N = n(), y = sum(y))
#Fit the model
model = brm(y|trials(N)~groups, data = modified_d, family = binomial())

From the model, we can extract posterior samples of each variant's rate

# .value column gives the posterior rates.
draws = tibble(groups = LETTERS[1:3], N=1) %>% 
  add_fitted_draws(model)

draws
 groups     N  .row .chain .iteration .draw .value
  <chr>  <dbl> <int>  <int>      <int> <int>  <dbl>
1 A          1     1     NA         NA  1004 0.107 
2 A          1     1     NA         NA   704 0.105 
3 A          1     1     NA         NA   377 0.103 
4 B          1     2     NA         NA  1465 0.134 
5 B          1     2     NA         NA    46 0.120 
6 B          1     2     NA         NA   324 0.124 
7 C          1     3     NA         NA  1623 0.0906
8 C          1     3     NA         NA  3421 0.0900
9 C          1     3     NA         NA   788 0.0868

Now, all that is needed is to compute our expected loss by using these draws to compute our loss function and averaging over the results. Here is how you could do it in R.

# Extract posterior estimates for each variant

A = filter(draws, groups=='A')$.value
B = filter(draws, groups=='B')$.value
C = filter(draws, groups=='C')$.value

# Loss function in R code
loss = function(choose, ignore_1, ignore_2) pmax(pmax(ignore_1, ignore_2) - choose, 0)

# Compute the expected loss over posterior samples
# Loss if I chose A
loss_A = mean(loss(A, B, C))
>  0.019 # May be different from run to run
# If I chose B
loss_B = mean(loss(B, A , C))
> 2.3e-05
# If I chose C
loss_C = mean(loss(C, A, B))
>0.035465

Now all we do is pick the variant which minimizes the loss. Variant B has lowest expected loss. If we chose B and we were wrong, then we estimate we lose out on a potential 0.0023% units to our metric. Compare this with A in which we lose out on a possible 2% to our metric. In fact, when I simulated the data, I made sure B had the largest rate. So our decision would be a good one in this case.

One thing I have not talked about is priors. If you do this approach, you may want to think about good priors to use for your model which is as much an art is it is a science. That being said, with enough data the priors matter less and less (especially for a simple experiment like this).