I'm relatively new to A/B split-testing and can't wrap my brain around the idea of z-scores.

I know that a z-score gives one an idea how statistically significant the result is that one gets from a split-test.

However, I found quite a few websites that explain A/B split-testing without talking much about sample sizes. These articles even suggested that the z-score itself would only show statistical significance if the datasets used to calculate it is large enough.

So I ran a one-tailed test. I made a modification to an element of a page and checked how the changes performed in terms of conversion rates. This was the result:

| Treatment   | Hits | Conv | %      | z-Score |
| Control     | 701  | 200  | 28.53% | -       |
| Treatment 1 | 699  | 228  | 32.62% | 1.66    |

Confidence (z-Score): 95.17%, improvement of conversion rate: 14.3%

However, I figured later that this result lacks two things:

  1. I didn't define a desired minimum improvement in percentage in my hypothesis, e. g. "Treatment 1 performs at least 20% better than Control"
  2. I didn't define a required sample size but just let the test run until a z-score >= 1.65 was reached (which I heard is a bad thing to do).

I then decided I want at least a 22% improvement of my conversion rate and that I needed 2,046 datasets to tell whether or not this goal was achieved (calculated based on this blog post).

The hypothesis was then:

Treatment 1 improves the conversion rate by at least 22% and I can tell with a probability of 95% that this change is not due to chance.

The final result (2,046 datasets) showed that Treatment 1 didn't improve anything really and that I can reject the hypothesis:

| Treatment   | Hits | Conv | %      | z-Score |
| Control     | 1020 | 305  | 29,90% | -       |
| Treatment 1 | 1026 | 308  | 30,02% | 0,06    |

What I don't understand is: What did the z-score of the 1,400 datasets tell me in plain English? What is it good for in this case?


1 Answer 1


Your original z-score tells you that, if the treatment had no effect, there would be less than a 5% chance of obtaining the scores you did obtain.

However that test depends on having a random sample of the population of interest. You didn't have a random sample because you deliberately selected one that would have a significant z-score.

Even if you did have a random sample, you would by definition expect in about 5% of cases to obtain scores indicating a positive effect when there is no such effect in the population.

As the blog post you refer to explains, increasing the sample size increases the power of the test, defined as one minus the probability of obtaining non-significant results in the sample given that there is a real treatment effect in the population. The post chooses a sample that results in a power of 95%. This is somewhat arbitrary, though. If testing is cheap you could go for 99% or even 99.9% and be more confident of not having missed a real effect.

In this case your original z-scores are not good for much because they are not randomly drawn. However, a random sample of an arbitrarily decided sample size, e.g. 1000, would still be useful, but (compared to a larger random sample) there would be a slightly greater risk of not capturing a real effect in the population.

  • $\begingroup$ Thanks for your answer. I think I get it now, even though this leads to a few more questions for me :) But I need to clarify one thing: You say "if testing is cheap you could go for 1% or 0.1%". By that you mean I should aim for a z-score that shows 99% or even 99.9% confidence? I'll try that next time, but I doubt I have enough traffic for that. To collect 2,046 datasets took me 33 days, which is a bit too long :) $\endgroup$
    – ckck
    Nov 1, 2012 at 3:21
  • $\begingroup$ Sorry, I've edited my answer. I meant that you could go for a power of 99% or 99.9%. I did not mean to suggest you should go for a z-score that shows higher confidence. As I said in the answer, selecting a sample on the basis of the z-score it produces is what caused you to have a non-random sample in the first place. $\endgroup$
    – Stuart
    Nov 1, 2012 at 10:14
  • $\begingroup$ There's nothing wrong with choosing a sample size that will result in 95% power. I just meant that the decision to take a bigger sample is always a trade-off between the extra cost (in money, time, server use etc.) and the benefit in terms of increased power. There is always a possibility that a random sample will fail to show an effect when there is a real effect in the population; that's the nature of sampling. You just need to reduce that risk to what you find an acceptable level, given the resources available for testing. $\endgroup$
    – Stuart
    Nov 1, 2012 at 10:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.