We sent three different emails to the same set of members (between the second and third emails, members who left the plan were removed and new eligible members were added). I don't have the member-level data. I have the following email open rates. How do I calculate the statistical differences between the email open rates?
Email 1 - 78 opened (out of 5,708); Email 2 - 161 opned (out of 5,708); Email 3 - 203 opned (out of 5,660);
If they opened the email or not is a Bernoulli experiment, so the total number opened could be modeled as having a binomial distribution. That is really assuming that the open probability is the same for all members, in the same campaign, but seems to be the best we could do. So what I would do is calculate (and then maybe draw) confidence intervals for the open probability, for the three campaigns. But your $n$'s are quite large, so just looking at the numbers there is quite clear the differences are real. But:
With your data and R we can do:
prop.test( c(78, 161, 201), c(5708, 5708, 5660))
3-sample test for equality of proportions without continuity
correction
data: c(78, 161, 201) out of c(5708, 5708, 5660)
X-squared = 56.068, df = 2, p-value = 6.684e-13
alternative hypothesis: two.sided
sample estimates:
prop 1 prop 2 prop 3
0.01366503 0.02820603 0.03551237
But the individual confidence intervals might be more informative. You can get them by:
binom.test(78, 5708)$conf.int
binom.test(161, 5708)$conf.int
binom.test(203, 5660)$conf.int
[1] 0.01081622 0.01702561
attr(,"conf.level")
[1] 0.95
> [1] 0.02406598 0.03283720
attr(,"conf.level")
[1] 0.95
> [1] 0.03117348 0.04104390
attr(,"conf.level")
[1] 0.95