Skip to main content
Cross Validated

Return to Answer

added 384 characters in body
Source Link
Amin Shn
Amin Shn
  • 765
  • 3
  • 14

where $p_{i}$ is the probability of opening the email and $y_{i}$ is zero or one indicating whether target i has opened the email. And you model thatthe probability with an intercept that changes according to the system being used:

You can see the full distribution estimated for $logit(p1)$ of system one ($alpha[1]$) and for $logit(p2)$ of system two ($alpha[2]$)(you can simply use $inv.logit()$ function to calculate the probability of opening the email for each). As you can see the model successfully estimated the true values (check the mean values $mu.vect$) and you can also observe the 95% CI which doesn't contain zero meaning the results are significant in frequentist terms. Note that the confidence interval associated with $alpha[1]$ is wider because you have less data on it but it doesn't matter as long as you can reasonably estimate the uncertainty (which we did) and see if you can live with that. Then increasing the sample size depends on how much risk are you willing to take.

Also you can easily expand this to incorporate different demographic groups and run the model over the whole dataset you have and give it a hierarchical structure to share information and reduce the uncertainty on the $alpha$ parameters you want to estimate without increasing the sample size.

where $p_{i}$ is the probability of opening the email. And you model that with an intercept that changes according to the system being used:

You can see the full distribution estimated for $logit(p1)$ of system one ($alpha[1]$) and for $logit(p2)$ of system two ($alpha[2]$)(you can simply use $inv.logit()$ function to calculate the probability of opening the email for each). As you can see the model successfully estimated the true values (check the mean values $mu.vect$) and you can also observe the 95% CI which doesn't contain zero meaning the results are significant in frequentist terms. Note that the confidence interval associated with $alpha[1]$ is wider because you have less data on it but it doesn't matter as long as you can reasonably estimate the uncertainty (which we did) and see if you can live with that. Then increasing the sample size depends on how much risk are you willing to take.

where $p_{i}$ is the probability of opening the email and $y_{i}$ is zero or one indicating whether target i has opened the email. And you model the probability with an intercept that changes according to the system being used:

You can see the full distribution estimated for $logit(p1)$ of system one ($alpha[1]$) and for $logit(p2)$ of system two ($alpha[2]$)(you can simply use $inv.logit()$ function to calculate the probability of opening the email for each). As you can see the model successfully estimated the true values (check the mean values $mu.vect$) and you can also observe the 95% CI which doesn't contain zero meaning the results are significant in frequentist terms. Note that the confidence interval associated with $alpha[1]$ is wider because you have less data on it but it doesn't matter as long as you can reasonably estimate the uncertainty (which we did) and see if you can live with that. Then increasing the sample size depends on how much risk are you willing to take.

Also you can easily expand this to incorporate different demographic groups and run the model over the whole dataset you have and give it a hierarchical structure to share information and reduce the uncertainty on the $alpha$ parameters you want to estimate without increasing the sample size.

Source Link
Amin Shn
Amin Shn
  • 765
  • 3
  • 14

You don't need any tests, just run a simple Bayesian logistic regression model:

$y_{i} \sim Binomial(1, p_{i})$

where $p_{i}$ is the probability of opening the email. And you model that with an intercept that changes according to the system being used:

$logit(p_{i}) = \alpha(System_{i})$

This is how you can fit this model using JAGS in R:

library(R2jags) # Package needed for Bayesian inference
library(boot) # Package contains the logit transform

# Some R code to simulate data 

T1 <- 160 #sample size for system 1
T2 <- 4100 #sample size for system 2
  
set.seed(123)

alpha_1 <- 0.8
alpha_2 <- 0.3

logit_p1 <- alpha_1 
logit_p2 <- alpha_2 


p1 <- inv.logit(logit_p1)
p2 <- inv.logit(logit_p2)


y1 <- rbinom(T1, 1, p1)
y2 <- rbinom(T2, 1, p2)

df1 <- data.frame(y = y1, system = 1)
df2 <- data.frame(y = y2, system = 2)
df_merged <- rbind(df1, df2)


# Jags code to fit the model to the simulated data
model_code <- "
model
{
  # Likelihood
  for (i in 1:I) {
    y[i] ~ dbin(p[i], k)
    logit(p[i]) <- alpha[system[i]]
  }
  
  # Priors
  
  for(j in 1:J){
  alpha[j] ~ dnorm(0.0,0.01)
  }

}
"

# Set up the data
model_data <- list(I = nrow(df_merged), 
                   y = df_merged$y, 
               system = df_merged$system,
                   k = 1,
                   J = 2)

# Choose the parameters to watch
model_parameters <- c("alpha")

# Run the model
model_run <- jags(
  data = model_data,
  parameters.to.save = model_parameters,
  model.file = textConnection(model_code)
)

print(model_run)

And this is the results:

pic

You can see the full distribution estimated for $logit(p1)$ of system one ($alpha[1]$) and for $logit(p2)$ of system two ($alpha[2]$)(you can simply use $inv.logit()$ function to calculate the probability of opening the email for each). As you can see the model successfully estimated the true values (check the mean values $mu.vect$) and you can also observe the 95% CI which doesn't contain zero meaning the results are significant in frequentist terms. Note that the confidence interval associated with $alpha[1]$ is wider because you have less data on it but it doesn't matter as long as you can reasonably estimate the uncertainty (which we did) and see if you can live with that. Then increasing the sample size depends on how much risk are you willing to take.