I have conducted A/B test on Newsletter send outs regarding title length over 10 week period. One group of people during those weeks always got long title, other short version of title. Newsletters did not differ in other things then title, content was same.

To check if in one particular case one of the newsletter in pair had higher Open Ratio I need to use test of proportions (t.test). The better result in one particular case might be due to better title text, not due to length of title, but other factors (for example more funny or more elegant: Short - "Buy with us", Long - "Need present for Christmas? Buy with us"). So how can I take into account repetitiveness of the test and check if one option was systematically better Open Rate then other?

Sample data:


NL_Data <- data.frame(Title= rep(c("Short", "Long"), 10),
                  Date = rep(as.Date("2017-01-01") + 7*1:10 , each = 2),
                  Open_Rate = c(runif(10, min = 0.02, max = 0.04), runif(10, min = 0.03, max = 0.05)),
                  Sample_Size = rep(c(350000, 500000), 10))

My idea is to calculate mean Open_Rate in each type of newsletter (with Long and Short title) and then do a t.test with sample size equal to sum of each send-out.

  • $\begingroup$ 10 weeks isn't very long, but you should try a paired t-test. It is an option in the t.test function of r. $\endgroup$ Commented Nov 20, 2017 at 8:44
  • 1
    $\begingroup$ t.test is not test of proportions, but test of means. What you need to use is prop.test. The ideal dataset of an AB test like that, when your target metric is Open Rate, should have been coded as 1 if emailed was opened (at least once) and coded as 0 if email was not opened. Open Ratio is a different thing and it's not a proportion even if it looks like one, If you send out 1 email and the user opens it twice then the open ratio is 2 and by default not a proportion. $\endgroup$
    – AntoniosK
    Commented Nov 20, 2017 at 8:53
  • $\begingroup$ Also, why did you stop the test after 10 weeks? Did you do any analysis in advance to prove that this time frame is enough to spot statistically significant differences in your target metric? $\endgroup$
    – AntoniosK
    Commented Nov 20, 2017 at 9:02
  • $\begingroup$ Some info here might help you : stackoverflow.com/questions/45361942/… , emailmarketingtipps.de/2012/05/14/… , ucanalytics.com/blogs/… $\endgroup$
    – AntoniosK
    Commented Nov 20, 2017 at 9:07
  • $\begingroup$ @JonnoBourne - I selected 10 weeks period as we would like to test many things, and 10 newsletters seemed like pretty fair time frame to test. How many newsletters do you recommend? $\endgroup$
    – AAAA
    Commented Nov 28, 2017 at 10:21

1 Answer 1


I propose to go away from the t-test and to the test of proportions given by prop.test. You need absolute numbers instead of shares for that:

NL_Data$Open <- round(NL_Data$Open_Rate*NL_Data$Sample_Size) #compute absolute numbers of opened emails

opened <- tapply(NL_Data$Open, NL_Data$Title, sum)  # total in all samples
sampled <- tapply(NL_Data$Sample_Size, NL_Data$Title, sum) # total in all samples
prop.test(opened, sampled) # test it via chi squared

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.