I'm relatively new to A/B split-testing and can't wrap my brain around the idea of z-scores.
I know that a z-score gives one an idea how statistically significant the result is that one gets from a split-test.
However, I found quite a few websites that explain A/B split-testing without talking much about sample sizes. These articles even suggested that the z-score itself would only show statistical significance if the datasets used to calculate it is large enough.
So I ran a one-tailed test. I made a modification to an element of a page and checked how the changes performed in terms of conversion rates. This was the result:
| Treatment | Hits | Conv | % | z-Score | |-------------|:----:|:----:|:------:|--------:| | Control | 701 | 200 | 28.53% | - | | Treatment 1 | 699 | 228 | 32.62% | 1.66 |
Confidence (z-Score): 95.17%, improvement of conversion rate: 14.3%
However, I figured later that this result lacks two things:
- I didn't define a desired minimum improvement in percentage in my hypothesis, e. g. "Treatment 1 performs at least 20% better than Control"
- I didn't define a required sample size but just let the test run until a z-score >= 1.65 was reached (which I heard is a bad thing to do).
I then decided I want at least a 22% improvement of my conversion rate and that I needed 2,046 datasets to tell whether or not this goal was achieved (calculated based on this blog post).
The hypothesis was then:
Treatment 1 improves the conversion rate by at least 22% and I can tell with a probability of 95% that this change is not due to chance.
The final result (2,046 datasets) showed that Treatment 1 didn't improve anything really and that I can reject the hypothesis:
| Treatment | Hits | Conv | % | z-Score | |-------------|:----:|:----:|:------:|--------:| | Control | 1020 | 305 | 29,90% | - | | Treatment 1 | 1026 | 308 | 30,02% | 0,06 |
What I don't understand is: What did the z-score of the 1,400 datasets tell me in plain English? What is it good for in this case?