3
$\begingroup$

We ran an AB test for four days, sending our clients the same emails but with different headings, in order to know which header logic is better (i.e. has has the highest Open Rate (OR) = openings/ sendings). All four days there were the same people in each group, i.e. everyone got all four emails. Though the sample sizes were large (almost a million entries each group), I feel that I have a dataset with only 4 entries for each group (OR for every day), as there is the mean OR and its distribution are of our interest. Am I right? And if so, how do I proceed with the test analyses? I feel that I need to have at least 30 entries for accurate results.

$\endgroup$
4
  • $\begingroup$ You should keep the OR in ratio form as x/n, as n gives info on the variance. BUT, you say four days, as in four different weekdays? Then there might be days effects, and if all mails with one heading went out the same days, how do you differentiate between day effects and header effects? You need to tell us details of the design! $\endgroup$ Jul 21 '20 at 15:14
  • 1
    $\begingroup$ @kjetilbhalvorsen Hi! Thank you very much for your answer and sorry I have missed it. $\endgroup$
    – Anat Manat
    Jul 28 '20 at 13:43
  • $\begingroup$ The same people get email every day? So you're telling me that the same person would exist in the data 4 times? $\endgroup$ Jul 28 '20 at 13:53
  • $\begingroup$ @DemetriPananos yes, please see the details in a comment below $\endgroup$
    – Anat Manat
    Jul 28 '20 at 21:18
1
$\begingroup$

You should keep the OR in ratio form as $x/n$, as $n$ gives info on the variance. BUT, you say four days, as in four different weekdays? Then there might be days effects, and if all mails with one heading went out the same days, how do you differentiate between day effects and header effects?

You need to tell us details of the design!

$\endgroup$
4
  • $\begingroup$ Every Tuesday we would test some logic in headers of emails, say, one group would get an email with an emoji in a header, and another group without it. We need to know whether emoji has any effect on OR. We did this 4 times. During the whole experiment every user sticked with his group and got all 4 letters. I am not sure whether I need to compare averages of ORs of each group or sum results of all four tests together and comapare sum(x)/n for each group, where x is a number of users, who opened at least once, n - all users in a group. $\endgroup$
    – Anat Manat
    Jul 28 '20 at 20:50
  • $\begingroup$ I have some doupts about the second option, because for me this feels the same as conducting this test just once. Isn't this just a point estimate and probability that the difference is only by accident is too high? $\endgroup$
    – Anat Manat
    Jul 28 '20 at 20:58
  • $\begingroup$ Any links to literature\ anything else would be greatly appreciated (I did an inference statistics course, but obviously missed the fundamentalsl) $\endgroup$
    – Anat Manat
    Jul 28 '20 at 21:06
  • $\begingroup$ So you gave the same treatment 4 times to each person? Probably it would have been better to give each person treatment A two days and treatment B the other two. Please add this additional detail on the design as an edit to the post! $\endgroup$ Aug 11 '20 at 23:07
1
$\begingroup$

I think because everyone is observed the same number of times, you could probably just get away with estimating the open rate for everyone and then doing something like a t-test.

I've simulated this approach (see below) assuming each person has a random effect as well (I think this is a fairly valid assumption, everyone has different email habits), and it seems like the results are approximately unbiased and have the correct false positive rate when the null is true. The question then becomes about power, which can also be simulated.

library(tidyverse)

#Set up the data
num_members<-100
num_times_obs<-4

#Randomize the members
group<-sample(c('A','B'), size = num_members, replace = T)
members<-tibble(members = 1:num_members, group)

#Construct random and fixed effects.
X <- model.matrix(~group, data = members)
Z <- model.matrix(~factor(members)-1, data = members)

#Simulate 1000 times doing a t test between estimated OR
results<-rerun(1000,{
  #Generate data
  beta <- c(qlogis(0.25), 0)
  gamma <- rnorm(ncol(Z), 0, 0.25) 
  eta <- X %*% beta + Z %*% gamma
  p<- plogis(eta)
  
  members$y<- rbinom(length(p),4, p)
  members$n<-4
  members$p_est = members$y/members$n
  
  p = t.test(p_est ~ group, data = members)$p.value
  mean_a = mean(filter(members, group=='A')$p_est)
  mean_b = mean(filter(members, group=='B')$p_est)
  
  tibble(A = mean_a,
         B = mean_b,
         p = p)
  
})  


p = map_dfr(results, ~.x) %>% 
  pull(p)


mean(p<0.05)
>>>0.045 (depending on random seed).

kjetil mentions possible effects of the day. That is certainly plausible, and so what you can do is do a logistic regression in order to adjust for days. Here is another example of how you might do that, this time using glmer since the same people are observed multiple times.

library(tidyverse)
library(lme4)

#Set up the data
num_members<-100
num_times_obs<-4

#Randomize the members
group<-sample(c('A','B'), size = num_members, replace = T)
members<-tibble(members = 1:num_members, group) %>% 
         crossing(days = factor(c('Fri','Sat','Sun','Mon')))

#Construct random and fixed effects.
X <- model.matrix(~group + days, data = members)
Z <- model.matrix(~factor(members)-1, data = members)

beta <- c(qlogis(0.25), 
          qlogis(0.27) - qlogis(0.25), 
          qlogis(.32) - qlogis(0.25),
          qlogis(0.15) - qlogis(0.25),
          qlogis(0.12)-qlogis(0.25))
gamma <- rnorm(ncol(Z), 0, 0.25) 
eta <- X %*% beta + Z %*% gamma
p<- plogis(eta)

members$y<- rbinom(length(p),1, p)

model = glmer(y~group + days + (1|member_id), data = members, family = binomial())

summary(model)

The approach will depend on what you're interested in adjusting for and what you think is appropriate.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.