# How to get Python Statsmodels to match Evan Miller's Famous AB Test Sample Size Calculator?

I'm trying to recreate this result. Which functions from statsmodels will do this?

Evan Miller Link: https://www.evanmiller.org/ab-testing/sample-size.html

I have tried the following:

import statsmodels.api as sm
from math import ceil

init_prop = 0.02
mde_prop = 0.03

effect_size = sm.stats.proportion_effectsize(
init_prop,
mde_prop
)

sample_size = sm.stats.tt_ind_solve_power(
effect_size=effect_size,
nobs1=None,
alpha=0.05,
power=0.8
)

print(f'{ceil(sample_size)} sample size required given power analysis and input parameters.')

# 3790 sample size required given power analysis and input parameters.


How are these so far off?

• I don't see a description of what evanmiller is doing. There are many hypothesis test comparing two proportions and several methods to compute exact or approximate power and sample size. Commented Feb 15, 2023 at 14:42
• For comparison: statsmodels sm.stats.samplesize_proportions_2indep_onetail(0.01, 0.02, 0.8) has nobs1=nobs2=3825 for two-sided test, the same as abtestguide.com/abtestsize Commented Feb 15, 2023 at 19:24
• You have to ask the awesome Miller :) For one-sided hypothesis, I get 3013 which is closer to Miller's. Note, one-sided requires doubling the alpha in the tail in sm.stats.samplesize_proportions_2indep_onetail(0.01, 0.02, 0.8, alpha=0.1) Commented Feb 15, 2023 at 19:36
• @josef just FYI, the following R code replicates what the Evan Miller's sample size calculator does, if it's in any way helpful: gist.github.com/uditgt/… (NB: I'm not the code author) Commented Feb 15, 2023 at 20:20
• @J-J-J Thanks, that makes the difference clear. I will edit my answer Commented Feb 15, 2023 at 21:01

Based on the comment by J-J-J to the question which shows how to replicate the result of the online calculator.

The difference between that online calculator and statsmodels samplesize_proportions_2indep_onetail function is in the assumption on the standard deviation of the statistic under the Null hypothesis. Statsmodels uses the pooled estimate (assuming proportions given by the alternative), while the online calculator assumes that the standard deviation is based on the proportion of the control.

When I add that option to the statsmodels code, I get the same result as the online calculator:

sm.stats.samplesize_proportions_2indep_onetail(0.01, 0.02, 0.8, null_var="prop2")
3292.2657351312073


Essentially, the null hypothesis assumes that the proportions are the same, but the methods differ in what that common value is assumed to be
(p1=0.03, p2=0.02 in the example)

• "prop2": prop1 = prop2 = p2 = 0.02
• "pooled": prop1 = prop2 = (p1 + p2) / 2 = 0.025

Note, the null hypothesis that the proportions are equal does not provide a condition on the common value. The common value is a "nuisance parameter" for the hypothesis test. Methods for testing this hypothesis differ in how they treat the nuisance parameter.

Which approximation?

There is a large number of hypothesis tests for comparing two Binomial proportions. Exact methods are usually very conservative in small samples. So many authors prefer approximate inexact methods that have better average size (rejections under the null).

Similar approximations will apply to sample size and power computation for hypothesis tests for two independent proportions. I applies also to other statistics like confidence intervals.

For example, statsmodels has the following inexact methods in test_proportions_2indep and similarly in the confidence interval function.

diff:

- 'wald',
- 'agresti-caffo'
- 'score' if correction is True, then this uses the degrees of freedom
correction nobs / (nobs - 1) as in Miettinen Nurminen 1985

ratio:

- 'log': wald test using log transformation
- 'log-adjusted': wald test using log transformation,
adds 0.5 to counts
- 'score': if correction is True, then this uses the degrees of freedom
correction nobs / (nobs - 1) as in Miettinen Nurminen 1985

odds-ratio:

- 'logit': wald test using logit transformation
- 'logit-adjusted': wald test using logit transformation,
adds 0.5 to counts
- 'logit-smoothed': wald test using logit transformation, biases
cell counts towards independence by adding two observations in
total.
- 'score' if correction is True, then this uses the degrees of freedom
correction nobs / (nobs - 1) as in Miettinen Nurminen 1985


In large samples, the answers will be very similar across methods.

statsmodels does not have a comprehensive list of power and sample size functions for those methods.

Power that uses proportion_effectsize is based on arcsin transformation, in imitation of the R pwr package. This is a variance stabilizing transformation.

The variance of proportions depends on the value of the proportion. This means that the variance differs between the null and alternative hypotheses.

Statsmodels has two functions samplesize_proportions_2indep_onetail and power_proportions_2indep for the asymptotic "z-test" for the null hypothesis that two proportions are the same. The resulting power and sample size will be only approximate if any of the other test methods are used that maintain average size.

(Aside: I am not a fan of "exact" methods. They usually rely on very strong distributional assumptions and cannot be made robust to misspecification.)

• Hi Josef, thanks very much for the detail you provided here. 'null_var' is not a kwarg of that function as I see it - could you share the changes you made to that function? Thanks
– mtn
Commented Feb 15, 2023 at 22:33
• github.com/statsmodels/statsmodels/pull/8676/files But I would prefer a better name for the keyword. Commented Feb 16, 2023 at 3:30
• (I think using p2 as common proportion is not really appropriate for power or sample size computation, but I don't remember the theory well enough.) Commented Feb 16, 2023 at 3:39
• Thanks a bunch Josef!
– mtn
Commented Feb 16, 2023 at 4:09