Sample proportion vs mean proportion for sample size estimation Context
I would like to estimate the sample size needed for an experiment. I’m testing a feature on a website and would like to detect a significant change between different variants .
One control and two treatment .
I’m new to this and want to keep this simple for the sake understanding tbe concept…
My parameters are : 0.05 alpha , 0.8 power.
Now if I know that tbe Click through rate of the original feature is 1% is this a mean or a proportion . I’m guessing this is the latter .
If I think the proportion for the second proportion will be 1.05% (i.e 5% minimal detectable effect) am I dealing with a case of sample size estimation for comparison of two proportions ?
Would the following estimation formula be adequate for such a test ?
n = (Zα/2+Zβ)^2 * (p1(1-p1)+p2(1-p2)) / (p1-p2)^2
Goal
Understand if I have the right approach?
Is it right to view the CTR as a proportion instead of mean ? If so when is mean the right choice ?
Is it a good formula for sample size estimation ? Are there more robust ones ? Is it a binomial distribution since one can only click or not click?
If my formula provides N samples and I would like to launch 15 variants does that mean I need 15xN samples ?
 A: First, to make sure I understand this correctly. You're looking at click through rates (CTR). Currently, the rate is about 1 percent. (This is a proportion.) You're interested in detecting an increase of 5 percent on this base rate of 1 percent, so 1.05 percent.
In order to maximize statistical power, I've made this a one-sided test. What I'm assuming is that you're really only interested in treatments if they improve CTR's compared to the status quo, i.e. the control group. Representing the proportion of the treatment group as $p_t$ and the proportion of the control group as $p_c$ this gives us the following:
$$
H_0: p_t = p_c \\
H_1: p_t > p_c
$$
The required sample is as follows where $n_c = n_t$, i.e. you take the same sample size for the treatment and control group:
$$
n_t = (p_t \times (1 - p_t) + p_c \times (1 - p_c)) \times (\frac{z_{1 - \alpha} + z_{1 - \beta}} {p_t - p_c})^2
$$
In the formula above, $\beta$ = 1 - power and $\alpha$ is the desired significance level, often 0.05. 80 percent is often taken as a rule of thumb for power though in some cases you might want more power. (More power just requires a larger sample size.) In this case, 80 percent statistical power means that if the treatment group does in fact achieve a 5 percent increase on the CTR compared to the control group, for hypothetical repeated experiments, in 80 percent of these identical experiments, we would detect this effect as statistical significant.
If we plug the proportions 0.01 for $p_c$ and 0.0105 for $p_t$ and 0.05 for $\alpha$ and 0.2 for $\beta$ into the formula above this gives us a required sample size of 501,771 for the treatment group and 501,771 for the control group.
In R:
p_c <- 0.01
p_t <- 0.0105
alpha <- 0.05
beta <- 0.20

(p_t*(1-p_t)+p_c*(1-p_c))*((qnorm(1-alpha)+qnorm(1-beta))/(p_t - p_c))^2

In Python:
from scipy.stats import norm

p_c = 0.01
p_t = 0.0105
alpha = 0.05
beta = 0.20

(p_t*(1-p_t)+p_c*(1-p_c))*((norm.ppf(1-alpha)+norm.ppf(1-beta))/(p_t - p_c))**2

The power.prop.test function in R returns a similar result.
power.prop.test(p1 = 0.01,
                p2 = 0.0105,
                sig.level = 0.05,
                power = 0.8,
                alternative = "one.sided")

If you change the alternative to "two.sided", you'll see that you would need a sample of 637,010 for each group to achieve the same power for a two-sided test.
Now, presuming that for each of your fifteen treatments, you want to test the treatment against the control, you'd need a sample of 16 (15 treatment groups plus one control group) x 501,771, so roughly a sample of 8 million.
Now, generally, you achieve maximum power by allotting the sample equally between the control and treatment group, but in your case, since you have multiple treatment groups each being tested against the same control group, you could reduce your total sample size by increasing the size of your control group. The standard error of the difference in proportions is given as follows:
$$ \sqrt{\frac{p_t \times (1 - p_t)}{n_t} + \frac{p_c \times (1 - p_c)}{n_c} } $$
By increasing the sample size for the control group, you decrease the standard error for the difference in proportions for each of your fifteen tests. The optimal size here for the control group would be something around 4 times larger than the treatment groups.
A modified version of the above formula includes $\kappa$ which equals $n_c / n_t$
$$
n_t = (p_t \times (1 - p_t) + \frac{p_c \times (1 - p_c)}{\kappa}) \times (\frac{z_{1 - \alpha} + z_{1 - \beta}} {p_t - p_c})^2
$$
Here, for your setup, you'd only need sample sizes of 318,149 for the treatment groups and a sample of 318,149 x 4 for the control group.
p_c <- 0.01
p_t <- 0.0105
alpha <- 0.05
beta <- 0.20
kappa <- 4
    
(p_t*(1-p_t)+(p_c*(1-p_c))/kappa)*((qnorm(1-alpha)+qnorm(1-beta))/(p_t - p_c))^2

This would get you close to 6 million samples needed. That said, optimal allocation ratios are a big topic. For more, you could start here and the reference therein.
Edit
Given that the formula's above rely on the Central Limit Theorem and are appropriate for large sample sizes, these are the same formula's you would use to compare two means. For a reference, see Sample Size Calculations in Clinical Research by Choa et. al. chapters 3 and 4.
Also, if you're doing 15 tests, you probably need to start thinking about correcting for false positives. If the null was true for each treatment, you'd still have a 75 percent chance of rejecting the null for at last one treatment.
