2
$\begingroup$

I would like to use the following power analysis formula into a python notebook

https://clincalc.com/stats/samplesize.aspx

Any idea which python package would replicate the same results ?

Edit:

I tried the following function from statsmodel module but can't seem to get anything close to the numbers in the webpage. Also , the formula in the webpage doesn't require a standard deviation . Why ?

statsmodels.stats.power.tt_ind_solve_power¶
$\endgroup$
4
  • 1
    $\begingroup$ Its easy to code these up yourself. Do you have a particular study in mind? I can provide the formulae necessary $\endgroup$ Apr 12, 2022 at 1:24
  • $\begingroup$ @DemetriPananos Im performing an A/B test for CTR on a button in a page and will use one control and 3 experiments. My base CTR is 0.5 % and anticipated CTR is 0.55% (10%MDE). Power = 0.8, Alpha = 0.05. I want to toggle between one-tail vs two-tail although im leaning towards one-tail. Hopefully the formula can help me find out minimum sample size needed providing me the same results as in the webpage provided above $\endgroup$ Apr 12, 2022 at 1:27
  • 1
    $\begingroup$ You don't need to enter the standard deviation because it's a function of the proportions, i.e. sd = sqrt(p * (1 - p)) $\endgroup$
    – num_39
    Apr 12, 2022 at 8:52
  • $\begingroup$ @num_39 i dont get the same results. Also the alternative function for value larger doesnt seem to work $\endgroup$ Apr 12, 2022 at 12:40

3 Answers 3

4
$\begingroup$

I think what's confusing you here is that the power function in the statsmodels in Python takes as an input Cohen's d for the effect size. Cohen's d scales the effect size in terms of pooled standard deviation. For your problem, you need to add the variance of the control and sample groups divided by two and then take the square root. This gives you Cohen's d.

I've modified the code from here to address you problem. You can change the input to 'two.sided' instead of 'larger' to see the required sample for the two-sided test.

p_c = 0.01
p_t = 0.0105

v_c = p_c * (1 - p_c)
v_t = p_t * (1 - p_t)
  
# calculate the pooled standard deviation 
# Cohen's d)
s = sqrt((v_c + v_t) / 2)
  
# calculate the effect size
d = (p_t - p_c) / s
print(f'Effect size: {d}')
  
# factors for power analysis
alpha = 0.05
power = 0.8
  
# perform power analysis to find sample size 
# for given effect
obj = TTestIndPower()
n = obj.solve_power(effect_size=d, alpha=alpha, power=power, 
                    ratio=1, alternative='larger')
  
print('Sample size/Number needed in each group: {:.3f}'.format(n))

Again, you'll see that the required sample size is 501771.

$\endgroup$
4
  • $\begingroup$ amazing! so in theory, both calculations achieve the same result but with a slight different approach? Im guessing that the ttindpower is more suitable for two means instead of two proportions ? $\endgroup$ Apr 12, 2022 at 13:04
  • $\begingroup$ what is the difference between smaller and larger? im guessing smaller is if the incidence is a decrease from the base and larger is the opposite ? $\endgroup$ Apr 12, 2022 at 13:06
  • 1
    $\begingroup$ Yes, larger is for a one-sided test that the treatment effect is > 0. Smaller would be for the one-sided test that the treatment effect is < 0. Here, though, if you try to use 'smaller' with a positive estimate for cohen's d, it won't converge. $\endgroup$
    – num_39
    Apr 12, 2022 at 14:22
  • 1
    $\begingroup$ Yes, it's the same approach except for the assumption of similar variance for cohen's d. In this case, though, you get the same sample size either way. $\endgroup$
    – num_39
    Apr 12, 2022 at 14:27
3
$\begingroup$

Because you're using 4 variants, this is a good case to just simulate power. I'm finding that typical power calculations actually give smaller power when we simulate the experiments, which to me is a good reason to ignore formulae and calculators and just do it yourself.

So here is a small R script to do just that. I'm going to simulate 4 exposures, three of which have a CTR of 0.005 and one of which has a CTR fo 0.0055. I'll estimate the CTR after simulating the data and determine how frequently I reject the null as a function of the sample size

library(tidyverse)

sim<-function(N){
  r = replicate(10000, {
  p <- c(0.005, 0.005, 0.005, 0.0055)
  y <- rbinom(4, N, p)
  fit <- glm(cbind(y, N-y) ~ factor(1:4), family = binomial())
  s <- summary(fit)
  p.val <- s$coefficients['factor(1:4)4', 'Pr(>|z|)']
  
  p.val<0.05
})
  
  mean(r)
}

d = tibble(N = seq(300000, 400000, 10000))%>% 
  mutate(
    pwr = map_dbl(N, sim)
  )

d %>% 
  ggplot(aes(N, pwr))+
  geom_line()+
  scale_y_continuous(labels = scales::percent)+
  scale_x_continuous(labels = scales::comma)+
  labs(x='Sample Size (Per Group)',
       y = 'Power')



Show below is the estimated power curve for a two tailed test. The code is easily edited to account for a one tailed test

enter image description here

Since the CTR is so small and proposed difference to detect is also very small, it looks like you're going to need 1.3 million users for this experiment.

$\endgroup$
4
  • $\begingroup$ This page shows you how to do something similar in Python: geeksforgeeks.org/introduction-to-power-analysis-in-python $\endgroup$
    – num_39
    Apr 12, 2022 at 8:59
  • $\begingroup$ Thanks for the proposed answer. Any chance you know python? I don't use R . $\endgroup$ Apr 12, 2022 at 12:14
  • $\begingroup$ @num_39 yea i've mentioned that function in my question but it uses a slightlly different approach than the formula I need to have and I'm not getting similar results at all. $\endgroup$ Apr 12, 2022 at 12:15
  • $\begingroup$ +1, but in this case I find that typical power calculations give the same result for 80% power. $\endgroup$
    – J-J-J
    Mar 2 at 9:47
3
$\begingroup$

​The initial implementation of power and sample size calculation followed Cohen's effect size tradition, with examples as @num_39 shows.

Recent versions of statsmodels provide more specialized function for binomial proportions and poisson rates.

Example:

https://www.statsmodels.org/dev/generated/statsmodels.stats.proportion.samplesize_proportions_2indep_onetail.html https://www.statsmodels.org/dev/generated/statsmodels.stats.proportion.power_proportions_2indep.html

from statsmodels.stats.proportion import samplesize_proportions_2indep_onetail
​
prop1 = 0.0105
prop2 = 0.01
diff = prop1 - prop2
power = 0.8
samplesize_proportions_2indep_onetail(diff, prop2, power, ratio=1, alpha=0.05, value=0, alternative='larger')
501772.20733602205

The difference between the effect size approach as num_39 showed and the distribution specific approach is in the assumption on the variance.

Cohen's d approach includes the standard deviation and assumes that the standard deviation is the same under the null and under the alternative hypothesis.

However, for GLM families like Binomial and Poisson, the variance depends on the mean, so the variance will be different in the alternative hypothesis from the variance under the null. The distribution specific power and sample size functions take this into account.

This is mainly relevant if there are large effect sizes and smaller sample sizes, otherwise the approximation based on effect size computation is close to the distribution specific results.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.