I'm trying to understand the reasoning by choosing a specific test approach when dealing with a simple A/B test - (i.e. two variations/groups with a binary respone (converted or not). As an example I will be using the data below
Version Visits Conversions A 2069 188 B 1826 220
The top answer here is great and talks about some of the underlying assumptions for z, t and chi square tests. But what I find confusing is that different online resources will cite different approaches, and you would think the assumptions for a basic A/B test should be pretty much the same?
- For instance, this article uses z-score:
- This article uses the following formula (which I'm not sure if it's different from the zscore calculation?):
- This paper references the t test(p 152):
So what arguemnts can be made in favor of these different approaches? Why would one have a preference?
To throw in one more candidate, the table above can be rewritten as a 2x2 contingency table, where Fisher's exact test (p5) can be used
Non converters Converters Row Total Version A 1881 188 2069 Versions B 1606 220 1826 Column Total 3487 408 3895
But according to this thread fisher's exact test should only be used with smaller sample sizes (what's the cut off?)
And then there's paired t and z tests,f test (and logistic regression, but I want to leave that out for now)....I feel like I'm drowning in different test approaches, and I just want to be able to make some kind of argument for the different methods in this simple A/B test case.
Using the example data I'm getting the following p-values
https://vwo.com/ab-split-test-significance-calculator/ gives a p-value of 0.001 (z-score)
http://www.evanmiller.org/ab-testing/chi-squared.html (using chi square test) gives a p-value of 0.00259
And in R
fisher.test(rbind(c(1881,188),c(1606,220)))$p.valuegives a p-value of 0.002785305
Which I guess are all pretty close...
Anyway - just hoping for some healthy discussion on what approaches to use in online testing where sample sizes are usually in the thousands, and response ratios are often 10% or less. My gut is telling me to use chi-square, but I want to be able to answer exactly why I'm choosing it over the other multitude of ways to do it.