SO in a sales setting where people can either buy or not buy (i.e. binary response), say I have a treatment that I know (or believe strongly) to be effective. I want to give it to as many people as possible, but I need to keep a control to determine what the actual lift is with some significance. The control group essentially means lost revenue, so I want to keep it as small as possible, but I need to be able to quantify the effect of the treatment. What is the approach for this? I found something called the Fleiss score but I haven't found this approach in other places.

This seems like a pretty common case, for instance if you want to give medication to as many patients as possible, but keep a minimum sized control group to verify the effect.

So some parameters as an example: 100K customers. Average baseline conversion rate is 10%. I want to be able to measure improvement of 1% absolute (10% relative, so 11%+) in conversion rate with 95% confidence and 80% power. What's the smallest I can make my control?

Using an A/B calculator the recommended sample size is about 14K per variation. So one approach would be to make the control 14K and let the rest receive treatment...but surely one can do better than that, given that the confidence interval will be much narrower for the test group if I increase the size to 86K? If there a commonly used approach for determining the smallest possible control?


1 Answer 1


Old question but recently worked on this and was surprised that no one had answered this! It's not really that hard. The rest of this follows from Chow page 61 (if you care about the proof). The sample size needed in treatment and control at a signficance level $\alpha$ and achieving power $1-\beta$ is: $$\frac{\epsilon-\delta}{\sigma\sqrt{\frac{1}{n_{t}} + \frac{1}{n_c}}}-z_{\alpha} = z_{\beta}$$ You can solve this for the the sample size of test, assuming that $n_t = T - n_c$: $$ n_c (1-\frac{n_c}{T})= \frac{(z_{\alpha} + z_{\beta})^2\sigma^2}{(\epsilon - \delta)^2} $$You get the maximum power when $n_c = n_t$ (i.e., a 50-50 split between test and control). So what you want to do is choose a power you are comfortable with plug it into the RHS of the above equation and then back out a value of $n_c$ that you get. For you this is $z_{1-\beta} = 0.84$, where $\beta=0.8$ and $z_{\alpha} = 1.96$ for $\alpha=0.95$ and $(\epsilon - \delta)/\sigma) = 1/\sigma$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.