I would like to estimate the sample size needed for an experiment. I’m testing a feature on a website and would like to detect a significant change between different variants . One control and two treatment .

I’m new to this and want to keep this simple for the sake understanding tbe concept…

My parameters are : 0.05 alpha , 0.8 power.

Now if I know that tbe Click through rate of the original feature is 1% is this a mean or a proportion . I’m guessing this is the latter .

If I think the proportion for the second proportion will be 1.05% (i.e 5% minimal detectable effect) am I dealing with a case of sample size estimation for comparison of two proportions ?

Would the following estimation formula be adequate for such a test ?

n = (Zα/2+Zβ)^2 * (p1(1-p1)+p2(1-p2)) / (p1-p2)^2 Goal Understand if I have the right approach? Is it right to view the CTR as a proportion instead of mean ? If so when is mean the right choice ? Is it a good formula for sample size estimation ? Are there more robust ones ? Is it a binomial distribution since one can only click or not click? If my formula provides N samples and I would like to launch 15 variants does that mean I need 15xN samples ?


1 Answer 1


First, to make sure I understand this correctly. You're looking at click through rates (CTR). Currently, the rate is about 1 percent. (This is a proportion.) You're interested in detecting an increase of 5 percent on this base rate of 1 percent, so 1.05 percent.

In order to maximize statistical power, I've made this a one-sided test. What I'm assuming is that you're really only interested in treatments if they improve CTR's compared to the status quo, i.e. the control group. Representing the proportion of the treatment group as $p_t$ and the proportion of the control group as $p_c$ this gives us the following:

$$ H_0: p_t = p_c \\ H_1: p_t > p_c $$

The required sample is as follows where $n_c = n_t$, i.e. you take the same sample size for the treatment and control group:

$$ n_t = (p_t \times (1 - p_t) + p_c \times (1 - p_c)) \times (\frac{z_{1 - \alpha} + z_{1 - \beta}} {p_t - p_c})^2 $$

In the formula above, $\beta$ = 1 - power and $\alpha$ is the desired significance level, often 0.05. 80 percent is often taken as a rule of thumb for power though in some cases you might want more power. (More power just requires a larger sample size.) In this case, 80 percent statistical power means that if the treatment group does in fact achieve a 5 percent increase on the CTR compared to the control group, for hypothetical repeated experiments, in 80 percent of these identical experiments, we would detect this effect as statistical significant.

If we plug the proportions 0.01 for $p_c$ and 0.0105 for $p_t$ and 0.05 for $\alpha$ and 0.2 for $\beta$ into the formula above this gives us a required sample size of 501,771 for the treatment group and 501,771 for the control group.

In R:

p_c <- 0.01
p_t <- 0.0105
alpha <- 0.05
beta <- 0.20

(p_t*(1-p_t)+p_c*(1-p_c))*((qnorm(1-alpha)+qnorm(1-beta))/(p_t - p_c))^2

In Python:

from scipy.stats import norm

p_c = 0.01
p_t = 0.0105
alpha = 0.05
beta = 0.20

(p_t*(1-p_t)+p_c*(1-p_c))*((norm.ppf(1-alpha)+norm.ppf(1-beta))/(p_t - p_c))**2

The power.prop.test function in R returns a similar result.

power.prop.test(p1 = 0.01,
                p2 = 0.0105,
                sig.level = 0.05,
                power = 0.8,
                alternative = "one.sided")

If you change the alternative to "two.sided", you'll see that you would need a sample of 637,010 for each group to achieve the same power for a two-sided test.

Now, presuming that for each of your fifteen treatments, you want to test the treatment against the control, you'd need a sample of 16 (15 treatment groups plus one control group) x 501,771, so roughly a sample of 8 million.

Now, generally, you achieve maximum power by allotting the sample equally between the control and treatment group, but in your case, since you have multiple treatment groups each being tested against the same control group, you could reduce your total sample size by increasing the size of your control group. The standard error of the difference in proportions is given as follows:

$$ \sqrt{\frac{p_t \times (1 - p_t)}{n_t} + \frac{p_c \times (1 - p_c)}{n_c} } $$

By increasing the sample size for the control group, you decrease the standard error for the difference in proportions for each of your fifteen tests. The optimal size here for the control group would be something around 4 times larger than the treatment groups.

A modified version of the above formula includes $\kappa$ which equals $n_c / n_t$

$$ n_t = (p_t \times (1 - p_t) + \frac{p_c \times (1 - p_c)}{\kappa}) \times (\frac{z_{1 - \alpha} + z_{1 - \beta}} {p_t - p_c})^2 $$

Here, for your setup, you'd only need sample sizes of 318,149 for the treatment groups and a sample of 318,149 x 4 for the control group.

p_c <- 0.01
p_t <- 0.0105
alpha <- 0.05
beta <- 0.20
kappa <- 4
(p_t*(1-p_t)+(p_c*(1-p_c))/kappa)*((qnorm(1-alpha)+qnorm(1-beta))/(p_t - p_c))^2

This would get you close to 6 million samples needed. That said, optimal allocation ratios are a big topic. For more, you could start here and the reference therein.


Given that the formula's above rely on the Central Limit Theorem and are appropriate for large sample sizes, these are the same formula's you would use to compare two means. For a reference, see Sample Size Calculations in Clinical Research by Choa et. al. chapters 3 and 4.

Also, if you're doing 15 tests, you probably need to start thinking about correcting for false positives. If the null was true for each treatment, you'd still have a 75 percent chance of rejecting the null for at last one treatment.

  • $\begingroup$ Did you decide for one side as we expect the MDE to only increase? So I’m not sure whether the 5% would go both ways or not . It would be two sided ? Also , can you please elaborate on how increasing the control group and decreasing the sample size per treament is beneficial for lowering sample size ? $\endgroup$ Commented Apr 8, 2022 at 16:27
  • $\begingroup$ I've added some information above. I'm presuming that you're only interested in the treatments if they increase CTR in which case you achieve maximum power by testing this hypothesis with a one-sided test. That said, I don't know your application. If you really need a two-sided test, just say so and I'll modify this. $\endgroup$
    – num_39
    Commented Apr 8, 2022 at 18:25
  • $\begingroup$ I’m not sure if it’s one side or two sided but I’d appreciate your advice. The application is an ad and I’m trying to optimizes CTR $\endgroup$ Commented Apr 8, 2022 at 19:30
  • $\begingroup$ Then I'd suggest you choose a one-sided test as you really are only interested in finding treatments that increase your CTR. $\endgroup$
    – num_39
    Commented Apr 8, 2022 at 19:43
  • $\begingroup$ how would you translate the first formula on python $\endgroup$ Commented Apr 12, 2022 at 1:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.