What does the z-score mean in plain English (in A/B split testing)?

Question

I'm relatively new to A/B split-testing and can't wrap my brain around the idea of z-scores.

I know that a z-score gives one an idea how statistically significant the result is that one gets from a split-test.

However, I found quite a few websites that explain A/B split-testing without talking much about sample sizes. These articles even suggested that the z-score itself would only show statistical significance if the datasets used to calculate it is large enough.

So I ran a one-tailed test. I made a modification to an element of a page and checked how the changes performed in terms of conversion rates. This was the result:

| Treatment   | Hits | Conv | %      | z-Score |
|-------------|:----:|:----:|:------:|--------:|
| Control     | 701  | 200  | 28.53% | -       |
| Treatment 1 | 699  | 228  | 32.62% | 1.66    |

Confidence (z-Score): 95.17%, improvement of conversion rate: 14.3%

However, I figured later that this result lacks two things:

1. I didn't define a desired minimum improvement in percentage in my hypothesis, e. g. "Treatment 1 performs at least 20% better than Control"
2. I didn't define a required sample size but just let the test run until a z-score >= 1.65 was reached (which I heard is a bad thing to do).

I then decided I want at least a 22% improvement of my conversion rate and that I needed 2,046 datasets to tell whether or not this goal was achieved (calculated based on this blog post).

The hypothesis was then:

Treatment 1 improves the conversion rate by at least 22% and I can tell with a probability of 95% that this change is not due to chance.

The final result (2,046 datasets) showed that Treatment 1 didn't improve anything really and that I can reject the hypothesis:

| Treatment   | Hits | Conv | %      | z-Score |
|-------------|:----:|:----:|:------:|--------:|
| Control     | 1020 | 305  | 29,90% | -       |
| Treatment 1 | 1026 | 308  | 30,02% | 0,06    |

What I don't understand is: What did the z-score of the 1,400 datasets tell me in plain English? What is it good for in this case?

Answer 1

Your original z-score tells you that, if the treatment had no effect, there would be less than a 5% chance of obtaining the scores you did obtain.

However that test depends on having a random sample of the population of interest. You didn't have a random sample because you deliberately selected one that would have a significant z-score.

Even if you did have a random sample, you would by definition expect in about 5% of cases to obtain scores indicating a positive effect when there is no such effect in the population.

As the blog post you refer to explains, increasing the sample size increases the power of the test, defined as one minus the probability of obtaining non-significant results in the sample given that there is a real treatment effect in the population. The post chooses a sample that results in a power of 95%. This is somewhat arbitrary, though. If testing is cheap you could go for 99% or even 99.9% and be more confident of not having missed a real effect.

In this case your original z-scores are not good for much because they are not randomly drawn. However, a random sample of an arbitrarily decided sample size, e.g. 1000, would still be useful, but (compared to a larger random sample) there would be a slightly greater risk of not capturing a real effect in the population.