# Complexities of interpreting A/B testing power & significance calculations

Apologies up front for not being a statistician. I've been searching high and low and I cannot find a sufficiently detailed explanation how to interpret differences in A/B testing power and significance calculations.

Here's my predicament. Imagine I've run a test where I bucket users of a SaaS solution when they sign-up (could be using Optimizely, Launch Darkly, Google, etc.) Each user is either in the baseline or the (single) variant group. Based on their bucket, they see a different experience from which they can take a conversion action.

My results could look like:

Baseline: 213 conversions of 1642 bucketed users
Variant: 219 conversions of 1359 bucketed users


The tool for this experiment (Launch Darkly) uses a z-score calculation and spits out:

Baseline conversion: 13.0% +/- 1.4%
Variant conversion: 16.1% +/- 1.6% (24.2% change) 99.2% confidence


An Evan Miller Sample Size Calculator for measuring a 24.2% minimum detectable effect (α = 5%, β = 20%) shows:

1846 samples are needed per bucket.


An Evan Miller Chi Square Test of Independence calculation shows confidence intervals (confidence level = 95%):

Baseline conversion: 11.4% - 14.7%
Variant conversion: 14.3% - 18.2%
p = 0.0146


An Evan Miller Sequential Sampling calculation (one-sided) for a 13% baseline conversion and 24.2% minimum detectable effect (α = 5%, β = 20%) shows:

Baseline winning if: 571 total conversions
Variant winning if: 47 conversions ahead


Therefore, while Chi-square and z-score both declare a significant difference between baseline and variant populations with high confidence, the power calculation and sequential sampling data suggest we need more data to be collected. My conundrum is understanding how to rationalize and interpret these results. Happy to read content that already exists if it tackles this topic.

I've been told in the past that there exists a unique power calculation per statistical technique (z-score, g-score, Chi-square, etc.) to account for the underlying assumptions about distributions. Does that factor in?

There are lots of things going on here. I'll just reel some points off.

1. I don't know what significance test Launch Darkly is using, but it's hard for me to guess how some kind of $z$-test would be useful for binary data.

2. The sample-size calculator doesn't say what test you're expected to use (in order for the given sample size to supply the given power). So it's not very interpretable.

3. Power is a non-issue if you've rejected the null hypothesis: if you've rejected the null hypothesis, you can't have possibly have made a type-II error, and all power is good for is avoiding type-II errors. The point of a power calculation is to decide how much data to get in advance of collecting it, not to say something about the data once you already have it.

4. A $χ^2$-test of independence is a traditional way to test the null hypothesis of independence between the two dimensions of a 2D contingency table. However, it uses a normal approximation, which is no longer necessary in the computer age. You're better off with Fisher's exact test or Barnard's test.

5. Sequential sampling, also called sequential hypothesis-testing, is a method to decide whether to stop or continue an ongoing study. If you know you don't want to collect any more data, don't use sequential tests; use a more conventional method like a plain Fisher's exact test. On the other hand, if you're trying to decide whether to collect more data on the basis of the data you already have, use sequential tests and not ordinary, non-sequential tests.

I've been told in the past that there exists a unique power calculation per statistical technique (z-score, g-score, Chi-square, etc.) to account for the underlying assumptions about distributions.

That's basically right. A power calculation is pretty much just the inverse of a hypothesis test, so the power calculation has to match the test.

• Thank you for the detailed response. There's a lot of great content here. Almost all SaaS A/B testing frameworks claim to use a z-test or variant, and then "simplify" the result by saying how "close" you are to being able to reject the null hypothesis. To wrap up this question, are you suggesting that A/B results should be put through a Fisher's exact test or Barnard's test (instead of χ2) to see if the null hypothesis can be rejected? I don't follow the comment about normal approximations and the computer age. Sep 12, 2017 at 11:47
• Separate question: what are the best online calculators for Fischer's exact test, Barnard's test, and the a priori power calculation for each? Sep 12, 2017 at 11:51
• @ChrisF "To wrap up this question, are you suggesting that A/B results should be put through a Fisher's exact test or Barnard's test (instead of χ2) to see if the null hypothesis can be rejected?" — I think that if you want to test this sort of null hypothesis with this sort of data, it's better to use Fisher's or Barnard's than $χ^2$, yes. Sep 12, 2017 at 14:42
• "I don't follow the comment about normal approximations and the computer age." — The reason people used to use a $χ^2$-test even though it's only approximate is that a $χ^2$-test can be calculated by hand without too much difficulty, whereas Fisher's exact test for large datasets is only realistically calculated with a computer. Sep 12, 2017 at 14:43
• "what are the best online calculators for Fischer's exact test, Barnard's test, and the a priori power calculation for each?" — I don't know; I use R and Python for statistics rather than web applications. Sep 12, 2017 at 14:43