Email open-rate test We ran an AB test for four days, sending our clients the same emails but with different headings, in order to know which header logic is better (i.e. has has the highest Open Rate (OR) = openings/ sendings). All four days there were the same people in each group, i.e. everyone got all four emails. Though the sample sizes were large (almost a million entries each group), I feel that I have a dataset with only 4 entries for each group (OR for every day), as there is the mean OR and its distribution are of our interest. Am I right? And if so, how do I proceed with the test analyses? I feel that I need to have at least 30 entries for accurate results.
 A: You should keep the OR in ratio form as $x/n$, as $n$ gives info on the variance. BUT, you say four days, as in four different weekdays? Then there might be days effects, and if all mails with one heading went out the same days, how do you differentiate between day effects and header effects?
You need to tell us details of the design!
A: I think because everyone is observed the same number of times, you could probably just get away with estimating the open rate for everyone and then doing something like a t-test.
I've simulated this approach (see below) assuming each person has a random effect as well (I think this is a fairly valid assumption, everyone has different email habits), and it seems like the results are approximately unbiased and have the correct false positive rate when the null is true.  The question then becomes about power, which can also be simulated.
library(tidyverse)

#Set up the data
num_members<-100
num_times_obs<-4

#Randomize the members
group<-sample(c('A','B'), size = num_members, replace = T)
members<-tibble(members = 1:num_members, group)

#Construct random and fixed effects.
X <- model.matrix(~group, data = members)
Z <- model.matrix(~factor(members)-1, data = members)

#Simulate 1000 times doing a t test between estimated OR
results<-rerun(1000,{
  #Generate data
  beta <- c(qlogis(0.25), 0)
  gamma <- rnorm(ncol(Z), 0, 0.25) 
  eta <- X %*% beta + Z %*% gamma
  p<- plogis(eta)
  
  members$y<- rbinom(length(p),4, p)
  members$n<-4
  members$p_est = members$y/members$n
  
  p = t.test(p_est ~ group, data = members)$p.value
  mean_a = mean(filter(members, group=='A')$p_est)
  mean_b = mean(filter(members, group=='B')$p_est)
  
  tibble(A = mean_a,
         B = mean_b,
         p = p)
  
})  


p = map_dfr(results, ~.x) %>% 
  pull(p)


mean(p<0.05)
>>>0.045 (depending on random seed).

kjetil mentions possible effects of the day.  That is certainly plausible, and so what you can do is do a logistic regression in order to adjust for days.  Here is another example of how you might do that, this time using glmer since the same people are observed multiple times.
library(tidyverse)
library(lme4)

#Set up the data
num_members<-100
num_times_obs<-4

#Randomize the members
group<-sample(c('A','B'), size = num_members, replace = T)
members<-tibble(members = 1:num_members, group) %>% 
         crossing(days = factor(c('Fri','Sat','Sun','Mon')))

#Construct random and fixed effects.
X <- model.matrix(~group + days, data = members)
Z <- model.matrix(~factor(members)-1, data = members)

beta <- c(qlogis(0.25), 
          qlogis(0.27) - qlogis(0.25), 
          qlogis(.32) - qlogis(0.25),
          qlogis(0.15) - qlogis(0.25),
          qlogis(0.12)-qlogis(0.25))
gamma <- rnorm(ncol(Z), 0, 0.25) 
eta <- X %*% beta + Z %*% gamma
p<- plogis(eta)

members$y<- rbinom(length(p),1, p)

model = glmer(y~group + days + (1|member_id), data = members, family = binomial())

summary(model)


The approach will depend on what you're interested in adjusting for and what you think is appropriate.
