Statistical Significance of A/B Test for Binary outcome

My situation is I have a list of people that were included in a marketing campaign. Half the list got version 1 of an email and the other half got version 2. Then people either bought the product or didn't.

Lets say both lists had 200 people. And in list 1(version 1 email) 20 bought the product. and in list 2 24 people bought the product.

In absolute terms it seems the version 2 email is more successful. Can someone tell me how to determine statistical significance for this test? and additionally, if you know of any resources where I can read/learn more about the topic of knowing which test to run in which scenario.

Thanks

• chi square test? en.wikipedia.org/wiki/Chi-squared_test or just simple logistic regression with 0-1 for the campaign and see if it is significant Nov 12 '19 at 14:53
• I think you are looking into proportion test stat.ethz.ch/R-manual/R-devel/library/stats/html/prop.test.html Nov 12 '19 at 14:54
• prop.test for two proportions, if the correct=FALSE option is chosen, is equivalent to a chi-square test of association (also without continuity correction). Nov 12 '19 at 15:04
• @rep_ho thank you I will read about that. I understand the basics of logistic regression, how would I apply that for this case? The independent variable would be the the campaign version? Thanks! Nov 12 '19 at 15:10
• There's some description of logistic regression, confidence intervals for proportions, and chi-square test of association for a similar question here. Nov 12 '19 at 16:06

1 Answer

Comment. Results from a one-sided 'test of two proportions' in Minitab shows no significant difference. (The R procedure prop.test will give a similar result.)

The first P-value is for an approximate normal test. The second is for 'Fisher's exact test', which uses a hypergeometric distribution. (The R procedure fisher.test will give a similar result.)

[Each of the tests is discussed on several pages on this site. Search terms in 'single quotes' above for more discussion. Perhaps start with pages linked above or under 'Related' in the right margin.]

Test and CI for Two Proportions

Sample   X    N  Sample p
1       20  200  0.100000
2       24  200  0.120000

Difference = p (1) - p (2)
Estimate for difference:  -0.02
95% upper bound for difference:  0.0314395
Test for difference = 0 (vs < 0):
Z = -0.64  P-Value = 0.261

Fisher’s exact test: P-Value = 0.316


In R, the hypergeometric CDF function phyper returns Minitab's P-value:

phyper(20, 200,200, 44)
[1] 0.3160336