Apologies up front for not being a statistician. I've been searching high and low and I cannot find a sufficiently detailed explanation how to interpret differences in A/B testing power and significance calculations.
Here's my predicament. Imagine I've run a test where I bucket users of a SaaS solution when they sign-up (could be using Optimizely, Launch Darkly, Google, etc.) Each user is either in the baseline or the (single) variant group. Based on their bucket, they see a different experience from which they can take a conversion action.
My results could look like:
Baseline: 213 conversions of 1642 bucketed users Variant: 219 conversions of 1359 bucketed users
The tool for this experiment (Launch Darkly) uses a z-score calculation and spits out:
Baseline conversion: 13.0% +/- 1.4% Variant conversion: 16.1% +/- 1.6% (24.2% change) 99.2% confidence
An Evan Miller Sample Size Calculator for measuring a 24.2% minimum detectable effect (α = 5%, β = 20%) shows:
1846 samples are needed per bucket.
An Evan Miller Chi Square Test of Independence calculation shows confidence intervals (confidence level = 95%):
Baseline conversion: 11.4% - 14.7% Variant conversion: 14.3% - 18.2% p = 0.0146
An Evan Miller Sequential Sampling calculation (one-sided) for a 13% baseline conversion and 24.2% minimum detectable effect (α = 5%, β = 20%) shows:
Baseline winning if: 571 total conversions Variant winning if: 47 conversions ahead
Therefore, while Chi-square and z-score both declare a significant difference between baseline and variant populations with high confidence, the power calculation and sequential sampling data suggest we need more data to be collected. My conundrum is understanding how to rationalize and interpret these results. Happy to read content that already exists if it tackles this topic.
I've been told in the past that there exists a unique power calculation per statistical technique (z-score, g-score, Chi-square, etc.) to account for the underlying assumptions about distributions. Does that factor in?