2
$\begingroup$

I've been researching the deep underlying mathematics of A/B testing. I understand A/B tests to be Bernoulli trials (conversion / non conversion) and categorical data represented in a 2x2 contingency matrix.

Why and how is it that many blogs or websites recommend using a Z-test to measure whether the two sample frequencies are drawn from the same population (accept null hypothesis), or are significantly different meaning they're drawn from different populations (reject null hypothesis)? How can a Z-test even be applied here?

This Wikipedia page doesn't list Z-test for categorical data: http://en.wikipedia.org/wiki/List_of_analyses_of_categorical_data

I understand that a G-test or Chi-square test is suitable, but does not provide a confidence interval, only a p-value to compare to a significance level. Is this related?

$\endgroup$
2
$\begingroup$

Why and how is it that many blogs or websites recommend using a Z-test to measure whether the two sample frequencies are drawn from the same population (accept null hypothesis), or are significantly different meaning they're drawn from different populations (reject null hypothesis)? How can a Z-test even be applied here?

As a two-sample proportions test.

One advantage over a chi-square is you can do a one-tailed test.

This Wikipedia page doesn't list Z-test for categorical data: http://en.wikipedia.org/wiki/List_of_analyses_of_categorical_data

This one does, search for "Two-proportion z-test".

$\endgroup$
1
$\begingroup$

The chi-squared of a 2x2 A/B test has only (2-1)x(2-1) = 1 degree of freedom, so it's equivalent to a Z-test. (A chi-squared distribution comes from the sum of squared z-scores.) In this specific case, the critical value of the Z-test is the square root of the critical value of the chi-squared test.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.