I'm trying to understand the reasoning by choosing a specific test approach when dealing with a simple A/B test - (i.e. two variations/groups with a binary respone (converted or not). As an example I will be using the data below

Version  Visits  Conversions
A        2069     188
B        1826     220

The top answer here is great and talks about some of the underlying assumptions for z, t and chi square tests. But what I find confusing is that different online resources will cite different approaches, and you would think the assumptions for a basic A/B test should be pretty much the same?

  1. For instance, this article uses z-score:enter image description here
  2. This article uses the following formula (which I'm not sure if it's different from the zscore calculation?):

enter image description here

  1. This paper references the t test(p 152):

enter image description here

So what arguemnts can be made in favor of these different approaches? Why would one have a preference?

To throw in one more candidate, the table above can be rewritten as a 2x2 contingency table, where Fisher's exact test (p5) can be used

              Non converters  Converters  Row Total
Version A     1881            188         2069  
Versions B    1606            220         1826
Column Total  3487            408         3895

But according to this thread fisher's exact test should only be used with smaller sample sizes (what's the cut off?)

And then there's paired t and z tests,f test (and logistic regression, but I want to leave that out for now)....I feel like I'm drowning in different test approaches, and I just want to be able to make some kind of argument for the different methods in this simple A/B test case.

Using the example data I'm getting the following p-values

  1. https://vwo.com/ab-split-test-significance-calculator/ gives a p-value of 0.001 (z-score)

  2. http://www.evanmiller.org/ab-testing/chi-squared.html (using chi square test) gives a p-value of 0.00259

  3. And in R fisher.test(rbind(c(1881,188),c(1606,220)))$p.value gives a p-value of 0.002785305

Which I guess are all pretty close...

Anyway - just hoping for some healthy discussion on what approaches to use in online testing where sample sizes are usually in the thousands, and response ratios are often 10% or less. My gut is telling me to use chi-square, but I want to be able to answer exactly why I'm choosing it over the other multitude of ways to do it.


2 Answers 2


We use these tests for different reasons and under different circumstances.

  1. $z$-test. A $z$-test assumes that our observations are independently drawn from a Normal distribution with unknown mean and known variance. A $z$-test is used primarily when we have quantitative data. (i.e. weights of rodents, ages of individuals, systolic blood pressure, etc.) However, $z$-tests can also be used when interested in proportions. (i.e. the proportion of people who get at least eight hours of sleep, etc.)

  2. $t$-test. A $t$-test assumes that our observations are independently drawn from a Normal distribution with unknown mean and unknown variance. Note that with a $t$-test, we do not know the population variance. This is far more common than knowing the population variance, so a $t$-test is generally more appropriate than a $z$-test, but practically there will be little difference between the two if sample sizes are large.

With $z$- and $t$-tests, your alternative hypothesis will be that your population mean (or population proportion) of one group is either not equal, less than, or greater than the population mean (or proportion) of the other group. This will depend on the type of analysis you seek to do, but your null and alternative hypotheses directly compare the means/proportions of the two groups.

  1. Chi-squared test. Whereas $z$- and $t$-tests concern quantitative data (or proportions in the case of $z$), chi-squared tests are appropriate for qualitative data. Again, the assumption is that observations are independent of one another. In this case, you aren't seeking a particular relationship. Your null hypothesis is that no relationship exists between variable one and variable two. Your alternative hypothesis is that a relationship does exist. This doesn't give you specifics as to how this relationship exists (i.e. in which direction the relationship goes) but it will provide evidence that a relationship does (or does not) exist between your independent variable and your groups.

  2. Fisher's exact test. One drawback to the chi-squared test is that it is asymptotic. This means that the $p$-value is accurate for very large sample sizes. However, if your sample sizes are small, then the $p$-value may not be quite as accurate. As such, Fisher's exact test allows you to exactly calculate the $p$-value of your data and not rely on approximations that will be poor if your sample sizes are small.

I keep discussing sample sizes - different references will give you different metrics as to when your samples are large enough. I would just find a reputable source, look at their rule, and apply their rule to find the test you want. I would not "shop around", so to speak, until you find a rule that you "like".

Ultimately, the test you choose should be based on a) your sample size and b) what form you want your hypotheses to take. If you are looking for a specific effect from your A/B test (for example, my B group has higher test scores), then I would opt for a $z$-test or $t$-test, pending sample size and the knowledge of the population variance. If you want to show that a relationship merely exists (for example, my A group and B group are different based on the independent variable but I don't care which group has higher scores), then the chi-squared or Fisher's exact test is appropriate, depending on sample size.

Does this make sense? Hope this helps!

  • $\begingroup$ Thanks for the detailed answer! I'm going to go through it in detail - I'm sure I'll have a few questions! $\endgroup$
    – L Xandor
    Oct 28, 2015 at 5:48
  • $\begingroup$ Could you further explain how the chi-squared and Fisher exact test don't indicate the direction of an effect? If all inferential statistics tests provide a confidence level around whether two samples sets are drawn from different populations or the same population, then what is it about the mathematical theory that won't let you say the directional difference in mean values would hold (B group has higher score)? $\endgroup$
    – Chris F
    Jan 5, 2016 at 3:59
  • 2
    $\begingroup$ For clarity, chi-squared test and Fisher's exact test are doing the same thing but the p-value is calculated slightly differently. (It's an approximation under chi-squared and an exact calculation under Fisher's exact.) I'll address chi-squared and it will generalize to Fisher's. The issue here is the premise. "If all inferential statistics tests provides a confidence level around whether two samples are drawn from..." - that's not what the chi-squared test does. The null hypothesis for the chi-squared test is that there is no association and the alternative hypothesis... $\endgroup$
    – Matt Brems
    Jan 5, 2016 at 13:23
  • 2
    $\begingroup$ ...is that there is some association between the two categorical variables. You're merely testing for the existence of an association and not pre-specifying a certain direction. (There are some lesser-known statistics out there that DO specify a certain relationship, so it is possible; however this is not what the chi-squared test is designed to do.) To infer afterward that there is a particular directional relationship based on a p-value that was calculated under a different set of hypotheses designed to just test for the existence of an association would be a mistake. $\endgroup$
    – Matt Brems
    Jan 5, 2016 at 13:29
  • 1
    $\begingroup$ I have been strugling with this: What is the logical difference between a A/B test and a relationship (proportion) test? To me, they are intuitively the same: testing if one group associates with specific characteristics. $\endgroup$
    – unicorn
    Sep 14, 2020 at 8:41

For a 3 way test you usually use an ANOVA rather than 3 separate tests. Please also check on the Bonferroni correction before multiple testing. Please use this https://www.google.com/search?q=testing+multiple+means&rlz=1C1CHBD_enIN817IN817&oq=testing+multiple+means+&aqs=chrome..69i57j69i60l3j69i61j0.3564j0j7&sourceid=chrome&ie=UTF-8


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.