# Statistical difference in email open rates

We sent three different emails to the same set of members (between the second and third emails, members who left the plan were removed and new eligible members were added). I don't have the member-level data. I have the following email open rates. How do I calculate the statistical differences between the email open rates?

Email 1 - 78 opened (out of 5,708); Email 2 - 161 opned (out of 5,708); Email 3 - 203 opned (out of 5,660);

• Please tell us what you mean by "statistical difference."
– whuber
Feb 10, 2022 at 16:38
• If the open rates are significantly different between these emails (p-value, 95% CI). I was thinking Chi-Square test but the emails were sent to same group of people. So not sure if I can use Chi-square or G-statistics. Feb 10, 2022 at 17:15
• Would you have a summary of how many people opened each of the eight possible combinations of e-mails?
– whuber
Feb 10, 2022 at 17:21
• No, I just have the above-mentioned data. Thanks. Feb 10, 2022 at 17:23

If they opened the email or not is a Bernoulli experiment, so the total number opened could be modeled as having a binomial distribution. That is really assuming that the open probability is the same for all members, in the same campaign, but seems to be the best we could do. So what I would do is calculate (and then maybe draw) confidence intervals for the open probability, for the three campaigns. But your $$n$$'s are quite large, so just looking at the numbers there is quite clear the differences are real. But:

With your data and R we can do:

 prop.test( c(78, 161, 201),  c(5708,  5708, 5660))

3-sample test for equality of proportions without continuity
correction

data:  c(78, 161, 201) out of c(5708, 5708, 5660)
X-squared = 56.068, df = 2, p-value = 6.684e-13
alternative hypothesis: two.sided
sample estimates:
prop 1     prop 2     prop 3
0.01366503 0.02820603 0.03551237


But the individual confidence intervals might be more informative. You can get them by:

 binom.test(78, 5708)$$conf.int binom.test(161, 5708)$$conf.int
binom.test(203, 5660)\$conf.int
 0.01081622 0.01702561
attr(,"conf.level")
 0.95
>  0.02406598 0.03283720
attr(,"conf.level")
 0.95
>  0.03117348 0.04104390
attr(,"conf.level")
 0.95