Complexities of interpreting A/B testing power & significance calculations

Question

Apologies up front for not being a statistician. I've been searching high and low and I cannot find a sufficiently detailed explanation how to interpret differences in A/B testing power and significance calculations.

Here's my predicament. Imagine I've run a test where I bucket users of a SaaS solution when they sign-up (could be using Optimizely, Launch Darkly, Google, etc.) Each user is either in the baseline or the (single) variant group. Based on their bucket, they see a different experience from which they can take a conversion action.

My results could look like:

Baseline: 213 conversions of 1642 bucketed users
Variant: 219 conversions of 1359 bucketed users

The tool for this experiment (Launch Darkly) uses a z-score calculation and spits out:

Baseline conversion: 13.0% +/- 1.4%
Variant conversion: 16.1% +/- 1.6% (24.2% change) 99.2% confidence

An Evan Miller Sample Size Calculator for measuring a 24.2% minimum detectable effect (α = 5%, β = 20%) shows:

1846 samples are needed per bucket.

An Evan Miller Chi Square Test of Independence calculation shows confidence intervals (confidence level = 95%):

Baseline conversion: 11.4% - 14.7%
Variant conversion: 14.3% - 18.2%
p = 0.0146

An Evan Miller Sequential Sampling calculation (one-sided) for a 13% baseline conversion and 24.2% minimum detectable effect (α = 5%, β = 20%) shows:

Baseline winning if: 571 total conversions
Variant winning if: 47 conversions ahead

Therefore, while Chi-square and z-score both declare a significant difference between baseline and variant populations with high confidence, the power calculation and sequential sampling data suggest we need more data to be collected. My conundrum is understanding how to rationalize and interpret these results. Happy to read content that already exists if it tackles this topic.

I've been told in the past that there exists a unique power calculation per statistical technique (z-score, g-score, Chi-square, etc.) to account for the underlying assumptions about distributions. Does that factor in?

Answer 1

There are lots of things going on here. I'll just reel some points off.

1. I don't know what significance test Launch Darkly is using, but it's hard for me to guess how some kind of $z$-test would be useful for binary data.

2. The sample-size calculator doesn't say what test you're expected to use (in order for the given sample size to supply the given power). So it's not very interpretable.

3. Power is a non-issue if you've rejected the null hypothesis: if you've rejected the null hypothesis, you can't have possibly have made a type-II error, and all power is good for is avoiding type-II errors. The point of a power calculation is to decide how much data to get in advance of collecting it, not to say something about the data once you already have it.

4. A $χ^2$-test of independence is a traditional way to test the null hypothesis of independence between the two dimensions of a 2D contingency table. However, it uses a normal approximation, which is no longer necessary in the computer age. You're better off with Fisher's exact test or Barnard's test.

5. Sequential sampling, also called sequential hypothesis-testing, is a method to decide whether to stop or continue an ongoing study. If you know you don't want to collect any more data, don't use sequential tests; use a more conventional method like a plain Fisher's exact test. On the other hand, if you're trying to decide whether to collect more data on the basis of the data you already have, use sequential tests and not ordinary, non-sequential tests.

I've been told in the past that there exists a unique power calculation per statistical technique (z-score, g-score, Chi-square, etc.) to account for the underlying assumptions about distributions.

That's basically right. A power calculation is pretty much just the inverse of a hypothesis test, so the power calculation has to match the test.