Problem: Analyze an experiment that was developed for the iOS app, called “ios_referral_experiment”, that tests adding new links to the referral invite page. The experiment has 3 groups: the “control” group has no new link; the “tab_only” group has a new navigation tab that links to the page; and the “tab_settings” group has both a new navigation tab and a new link in the settings. What are the results from this experiment and which group would you recommend based on the results?

Data Parameters:

This table includes click and view events in the UI.

● user_id: unique id for the user

● exposed_time: time when the user was exposed to the experiment

● split_test_group: the split test group the user was assigned to events

● event_time: time when the event occurred

● event_type: the type of event that happened

○ “referral_page_view”: user views the page for sending referral invites

○ “referrer_page_invite_action”: users clicks the button to send a referral invite

● event_type_button: the type of button the user clicks for the event referrals

                             user_id           split_test_group exposed_time                             event_type     platform       event_type_button       Date                Time          
 b861f69cb669766cc0d52bc2279332c0:   414   control     :27274   Length:142123      referrer_page_invite_action:  8896   iOS:135067              :133227   Min.   :2017-06-02   Length:142123     
 5ac762996eb6e8932fd2140e77a7a870:   376   tab_only    :67381   Class :character   referrer_page_viewed       :133227   Web:  7056   Link       :  1816   1st Qu.:2017-07-21   Class :character  
 00c26d8255da64576a9e3c5a2c1271eb:   298   tab_settings:47468   Mode  :character                                                     SMS        :  1727   Median :2017-08-04   Mode  :character  
 2d4fdc1606ad722a93a98882f9ccf331:   236                                                                                             link       :  1226   Mean   :2017-08-02                     
 4296772f2a0ce768d25573863c82968c:   236                                                                                             email      :  1118   3rd Qu.:2017-08-17                     
 fe7a74a142b5774375ba64184d67a8e3:   212                                                                                             ShareDialog:  1062   Max.   :2017-09-01                     
 (Other)                         :140351                                                                                             (Other)    :  1947           

I would like to calculate based on each 'split_test_group' - which had the greatest influence on the number of 'referrer_page_invite_action' vs 'referrer_page_viewed'.


New data table based on count per group

split_test_group   |   event_type                  |     count

Control                referrer_page_invite_action       1892
Control                referrer_page_viewed              25382
tab_only               referrer_page_invite_action       4009
tab_only               referrer_page_viewed              63372
tab_settings           referrer_page_invite_action       2995
tab_settings           referrer_page_viewed              44473
  • $\begingroup$ Can you elaborate more on your set up? What do you want to learn from this experiment? What measurement you want to use to verify your hypothesis? Which variable in your data set corresponds to this measurement. I also do not understand the print out you pasted as data. $\endgroup$
    – Wassermann
    Jun 5 at 21:35
  • 1
    $\begingroup$ Thanks for further details. If I get it right you are interested in the set up which maximizes probability of sending the referral invite. I also assume your control group also has a possibility of sending referral invite otherwise having a control group wouldn't make sense for me (am I thinking right?). I think I would calculate in each group a fraction of users who sent invite and do three tests to compare the fractions against each other. The two tests against control group would be to show if any of the new set ups brings any benefits, and if this would be truth for both... $\endgroup$
    – Wassermann
    Jun 5 at 22:10
  • 1
    $\begingroup$ ... I would test both against each other to see if one of them is superior compared to the other. $\endgroup$
    – Wassermann
    Jun 5 at 22:11
  • 1
    $\begingroup$ I would compare first both treatment groups against the control to make sure there is significant effect in these groups at all. And once this is confirmed I would compare the two treatment groups against each other to see if there is significant difference between them (both treatments can be equally good). ANOVA can tell you only if the three groups are equal or not, but if they are not equal it will tell you nothing about how do they relate to each other. $\endgroup$
    – Wassermann
    Jun 5 at 22:36
  • 1
    $\begingroup$ I'm not clear on the outcome. Are you interested in determining which variant lead to more clicks to send referrals, or are you interested in determining which variant lead to more page views? Or maybe both? In any case, you could do a very simple Bayesian decision analysis on this. I can provide more details and a simulated example if you can clarify the outcome. $\endgroup$ Jun 5 at 22:56

Doing a joint optimization for two metrics is an interesting problem, but unfortunately a bit out my expertise. However, making a decision based on a single metric is straight forward enough. Here is a simulated example of how this might be done using a Bayesian lense.

Let's first simulate some data


N = 10000
groups = sample(LETTERS[1:3], size = N, replace = T)

# Generate fake data
X = model.matrix(~groups)
true_beta = c(qlogis(0.1), qlogis(0.12) - qlogis(0.1), qlogis(0.09) - qlogis(0.1))
true_p = plogis(X %*% true_beta)

# The observations
y = rbinom(N, 1, true_p)
d = tibble(groups, y)

In our fake experiment, we have allocated approximately 3300 people to one of 3 groups and measured a binary outcome. Here is a summary of the data.

  groups     N     y
  <chr>  <int> <int>
1 A       3385   359
2 B       3311   415
3 C       3304   297

To begin with, we need a loss function, essentially a function which tells us what I would stand to lose were I to make a decision. Each action, including the do nothing scenario (where we just keep the control group as the main exposure) is a decision, so we need to know what we would lose if that were the wrong decision.

We really don't want to hurt the rate/metric at which the binary outcome happens (whatever that rate/metric is). One loss function we might find interesting is the following

$$ \mathcal{L}(x, p_a, p_b, p_c) = \begin{cases} \operatorname{max}(\operatorname{max}(p_b, p_c) - p_a, 0) & \quad x=a \\ \operatorname{max}(\operatorname{max}(p_a, p_c) - p_b, 0) & \quad x=b\\ \operatorname{max}(\operatorname{max}(p_a, p_b) - p_c, 0) & \quad x=c \end{cases}$$

Here, $x$ is the variant we choose to implement, and $p_i$ is the rate for group $i$. The function $\mathcal{L}$ tells us how much we would stand to hurt our metric if we chose the wrong variant. So for example, let's say we choose variant $a$. In the case where $a$ truly is the superior variant ($p_a>p_b$ and $p_a>p_c$) then $\max{\max{p_b, p_c} - p_a, 0} = 0$ and so we lose out on a total 0 change to the metric. But suppose that $p_b>p_a$, then choosing $a$ means we lose out on $p_b-p_a$. Had we chosen $b$, our metric would have been higher, but since we didn't we lose out on that increase.

The thing is that $p_a$, $p_b$, and $p_c$ are all unknown, so we have to estimate them from our data. We can do that fairly easily using R and some tools for logistic regression (I am using Bayesian methods because it makes the decision process very easy as we will see).

Using the data created above, we can estimate a logistic regression with

# Bayesian models library

# Modify data so we have success/trials as our coliumns
modified_d = group_by(d, groups) %>% summarise(N = n(), y = sum(y))
#Fit the model
model = brm(y|trials(N)~groups, data = modified_d, family = binomial())

From the model, we can extract posterior samples of each variant's rate

# .value column gives the posterior rates.
draws = tibble(groups = LETTERS[1:3], N=1) %>% 

 groups     N  .row .chain .iteration .draw .value
  <chr>  <dbl> <int>  <int>      <int> <int>  <dbl>
1 A          1     1     NA         NA  1004 0.107 
2 A          1     1     NA         NA   704 0.105 
3 A          1     1     NA         NA   377 0.103 
4 B          1     2     NA         NA  1465 0.134 
5 B          1     2     NA         NA    46 0.120 
6 B          1     2     NA         NA   324 0.124 
7 C          1     3     NA         NA  1623 0.0906
8 C          1     3     NA         NA  3421 0.0900
9 C          1     3     NA         NA   788 0.0868

Now, all that is needed is to compute our expected loss by using these draws to compute our loss function and averaging over the results. Here is how you could do it in R.

# Extract posterior estimates for each variant

A = filter(draws, groups=='A')$.value
B = filter(draws, groups=='B')$.value
C = filter(draws, groups=='C')$.value

# Loss function in R code
loss = function(choose, ignore_1, ignore_2) pmax(pmax(ignore_1, ignore_2) - choose, 0)

# Compute the expected loss over posterior samples
# Loss if I chose A
loss_A = mean(loss(A, B, C))
>  0.019 # May be different from run to run
# If I chose B
loss_B = mean(loss(B, A , C))
> 2.3e-05
# If I chose C
loss_C = mean(loss(C, A, B))

Now all we do is pick the variant which minimizes the loss. Variant B has lowest expected loss. If we chose B and we were wrong, then we estimate we lose out on a potential 0.0023% units to our metric. Compare this with A in which we lose out on a possible 2% to our metric. In fact, when I simulated the data, I made sure B had the largest rate. So our decision would be a good one in this case.

One thing I have not talked about is priors. If you do this approach, you may want to think about good priors to use for your model which is as much an art is it is a science. That being said, with enough data the priors matter less and less (especially for a simple experiment like this).

  • $\begingroup$ WOW! This is incredible. I'm quite new with bayesian stats and trying to figure out how I can adjust my data to fit your logic but this is extremely helpful! Im going to play around with my data and see where I get and will update the post (have to read up a little). Thank you again! $\endgroup$
    – Dinho
    Jun 6 at 0:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.