I'm fairly new to statistics. After a lot of research on all the different statistical significance tests out there and how to do them, I wonder why the Z test is so common for a/b testing in the marketing industry.

Here's my reasoning: In a test, both the A version and B version could deviate from their expected values. And since the Z test compares the observed A & B combined values as a whole, versus the expected A & B combined values as a whole, than the Z test is only answering the question: "Does both my A version and B version deviate from what's expected?" It doesn't truly compare the B version to the A version.

Or am I fundamentally missing something about how the Z test works? Hopefully I'm phrasing the question clearly.

  • 2
    $\begingroup$ You seem to be confusing one-sample tests with two sample tests. $\endgroup$
    – Glen_b
    Aug 22 '17 at 0:58

A typical A/B test in marketing is fundamentally a test of equal proportions, and there are several ways to perform this test.

In a marketing campaign, a certain number of people are contacted or exposed to an impression, and of those a certain number will "convert," which often means purchase something, but can be some other evidence of engagement, such as creating an account or simply clicking on something. This basic framework holds true for direct mail campaigns, email campaigns, display and paid search advertisements, and so on.

To A/B test a marketing campaign, we divide the people who will are exposed into two groups, conventionally called A and B, and use a different message, appeal, call-to-action, or graphic design for the two groups. Which version an individual is exposed to is completely random. For each group then, we know the total number of people exposed to each version of the message, and we know the number of conversions for each group. The data can be tabulated like so:

                   |   A  |  B
   did not convert |  605 | 195
   converted       |  351 | 41

These same data could be displayed in an equivalent form:

           |   n  |   p
   Group A |  956 | 0.367
   Group B |  236 | 0.174

We are interested to know if the two groups have the same conversion rate (the proportion of conversions in each group) or if there is a statistically significant different in conversion rate. In the language of statistics, we need to perform a two-sided test of equal proportions. The null hypothesis is that the two groups have equal proportions, and the alternative hypothesis is that the "true" proportions are not equal.

There are three popular tests to consider: Fisher's exact test, Pearson's $\chi^2$ test, and the z-test on equal proportions. These vary mainly by the degree of approximation involved, and consequently by the total sample size required to before we meet the assumptions of the test. For a typical marketing campaign, we will have many thousands of impressions and conversions. The z-test is used because it is easy to interpret and makes it easy to talk about effect size and confidence intervals. In some cases, some of the counts will be small (less than 30) usually for the number of conversions. In those cases, the $\chi^2$ or Fisher's exact test will be used for formal hypothesis testing and reporting significance level.

In my experience, the z-test is most often used, because if the sample sizes are so small, the campaign isn't worth analyzing anyway. (Although I have seen certain clients apply Fisher's exact test because they had only a handful of conversion, each worth tens of thousands of dollars, and perhaps other big ticket items like sports cars would likewise need to analyze campaigns with a very small number of conversions.)

The z-test of equal proportions can be understood as modeling each group as a Binomial distribution:

$$ \begin{align} A & \sim \text{Binomial}(n_A, p_A) \\ B & \sim \text{Binomial}(n_B, p_B) \\ \Delta & = B - A \end{align} $$

But what is the distribution of an RV defined as the difference of two Binomial RVs? Well, it's actually a bit intractable, so we approximate it with a normal distribution. By dividing by the standard error of $\Delta$, we can obtain a $z$ statistic which has, under the null hypothesis and our approximation, a standard normal distribution. You can easily find the formulas for this standard error; what's important to remember is that it is the difference in proportions between the two groups that drives our test statistic. This makes it very easy and intuitive to reason about. With the other tests we gain a modicum of validity at small sample sizes, but lose some of this ease of interpretation.

  • $\begingroup$ Thanks for this--it's been really hard to find a good answer to this question that speaks to the marketing aspect. Could you say a bit more about this part: "the z-test is used because it is easy to interpret and makes it easy to talk about effect size and confidence intervals." Specifically, how is Fisher's exact test hard to interpret or how does it complicate talking about effect size and confidence intervals? $\endgroup$
    – Khashir
    Jan 31 '21 at 3:00

Totally marketing-naive, but in general you're testing if the difference between A and B can be attributed to random chance or not, i.e. is the difference significant? A chi-squared test tests whether a ratio is significantly different (e.g. ratio of visitors that click on a link) and a z-test tests whether means are significantly different. I believe you're slightly confused, in that both tests do directly compare the A version to the B version.

There are probably many more out there, but the discussion in this question might be helpful re: the logic of using the tests. A/B tests: z-test vs t-test vs chi square vs fisher exact test


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.