Say I am running an AB test (control = $A$ and test = $B$). My significance level is 5% and power is 80%. My null hypothesis is that $p_A = p_B$ vs $p_A \ne p_B$.

My historical conversion rate for the control group is $p_A = 20\%$. Say that I only care if my test yields a conversion rate greater or less than 10% (relatively) different than my control. That is I only care if $p_B > 22\%$ or $< 18\%$.

I use the following formula for sample size estimation:

$$N = \frac {2 \times [Z_{\alpha/2}\sqrt{2p(1-p)} + Z_{\beta}\sqrt{p_A(1-p_A) + p_B(1-p_B)}]^2}{(p_A - p_B)^2}$$ (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3409926/)

Where $p_A = 20\%$ and $p_B = 22\%$, $\alpha = 0.05$ and $\beta = 0.2$.

This indicates that I need to collect at least 13K observations to detect a difference of at least 10%.

Now say I run the test, collect 13K samples, and observe that $p_A = 20\%$ and $p_B = 22\%$. Now I can call the test and say that version $B$ the winner, with an average conversion rate of $22\%$.

If someone asked me to make a typical business projection assuming 22% conversion, I would likely add in lower and upper confidence intervals and make some rough projection (no different than if I didn't run the test and just used my original 20%).

Now what if that same person was to say "just to be safe, instead of assuming we'll always convert at 22% (on average), let's instead project assuming only a 21% conversion rate. And let's drop the confidence intervals, I just care about the average".

But wait a second - I only collected 13K samples which is enough to detect a 10% relative difference, i.e. I can detect an absolute conversion difference of 2% or greater, not less. Projecting at 21% assumes that my conversion difference is significant at 1%, but I didn't collect enough samples to be able to make this statement. To detect a difference of 1% I'd need to collect 52K samples. So clearly I cannot make a projection of 21%. But I can make a projection of 22%...This logic seems wrong.

Can someone help me square this inconsistency? How come I can safely project at 22% but not at 21%? What am I missing here?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.