2
$\begingroup$

I have been using Lehr's rule to estimate the required sample sizes for some experiments I have to run.

Lehr's rule states that we need:

$$ n = \frac{ 2\left(z(1-\alpha) + z(1-\beta)\right)^2 \sigma^2}{\Delta^2} $$

Where $ z(x) $ is the $z$-score at significance level $x$, $\sigma^2$ is the sample variance and $\Delta^2$ is the minimum detecable effect.

Some code to calculate this required sample size is below.

import numpy as np
import pandas as pd
from scipy.stats import ttest_ind, norm
from statsmodels.stats.power import TTestPower

# Function to calculate the required sample size
def calculate_sample_size(alpha, beta, sample_variance, mde):
    ## https://en.wikipedia.org/wiki/Power_of_a_test#Rule_of_thumb

    norm_inv_z5 = norm.ppf(1.0 - alpha/2)
    norm_inv_1_z6 = norm.ppf(1.0 - beta)
    K = 2 * (norm_inv_z5 + norm_inv_1_z6)**2
    num = K * sample_variance
    denom = mde ** 2
    sample_size = num / denom
    return int(np.ceil(sample_size))

alpha = 0.1  # Significance level for t-tests
beta = 0.2  # Power level
CvR = 0.1  # Conversion rate
sample_variance = CvR * (1 - CvR)  # Sample variance
MDE = 0.05  # Minimum detectable effect %
n_samples = calculate_sample_size(alpha, beta, sample_variance, CvR * MDE)  # Sample size

In this example, I want to calculate the sample size for an intervention on a website. I want to increase the conversion rate by 5%. Doing this gives me a sample size of roughly 50,000.

Using the same inputs I have found the following Statsmodels function to calculate the power of a t-test:

 effect_size = (CvR * MDE) / sample_variance 
 analysis = TTestPower()
 n_samples = int(np.ceil(analysis.solve_power(effect_size, power=1.0-beta,   nobs=None, alpha=alpha, alternative='two-sided')))

However, this method only requires ~3000 samples.

What explains the difference, and why?

$\endgroup$
2
  • $\begingroup$ I think this question should be reopened. Based on the comments there is still a difference to the samplesize computation for the 2 sample t-test and the difference is very large with a factor around 10. (But I cannot figure out where the difference comes from.) $\endgroup$
    – Josef
    Commented May 29 at 15:02
  • $\begingroup$ @Björn in a comment to the accepted answer below found the "bug". effect_size is defined as effect divided by standard deviation, the code here divides by variance, the square root is missing. $\endgroup$
    – Josef
    Commented May 30 at 14:07

2 Answers 2

5
$\begingroup$

The formula you give (what you call Lehr's formula) is for a two-sample t-test comparing two groups. In contrast the [statsmodels.stats.power.TTestPower function is for a one-sample t-test] (https://www.statsmodels.org/stable/generated/statsmodels.stats.power.TTestPower.html) (or test for paired differences), which will by default test the null hypothesis that within a single group the average is 0 (vs. the alternative that it's $\neq 0$).

Thus, the two cannot give the same result and a within group comparison will always be more efficient (it compares to a fixed given number, while a two sample comparison compares to a control group for which the mean is also unknown).

$\endgroup$
8
  • $\begingroup$ If I use (tt_ind_solve_power)[statsmodels.org/dev/generated/… then I still get about 90% fewer samples (5000, compared to 50000). $\endgroup$
    – Tom Kealy
    Commented May 27 at 13:39
  • 2
    $\begingroup$ Well, for effect size of 0.005 and SD of 0.1111, in R power.t.test(power=0.8, sig.level=0.1, sd=0.1111, delta=0.005, alternative = "two.sided") gets me n/arm of 6106 (N total=12212), so I so that sounds closer to statsmodels than your manual calculation, which probably has some error. Arguably, one might want to use something more appropriate for proportions / binary events (e.g. logistic regression), but will not change the order of magnitude of the sample size. $\endgroup$
    – Björn
    Commented May 27 at 14:14
  • $\begingroup$ Something still does not match up. For example if I use proportion power, then I get around 45000 observations per sample. samplesize_proportions_2indep_onetail(0.005, 0.1, 0.8, ratio=1) in statsmodels. $\endgroup$
    – Josef
    Commented May 29 at 15:08
  • 2
    $\begingroup$ Ah, I inserted the wrong SD above, that's the variance. I get more or less the same answer as you with power.prop.test(p1=0.1, p2=0.095, sig.level = 0.1, alternative = "two.sided", power=0.8) gives 43521 per group and power.t.test(power=0.8, sig.level=0.1, sd=sqrt(0.1*(1-0.1)), delta=0.005, alternative = "two.sided") gives 44515 per group. $\endgroup$
    – Björn
    Commented May 29 at 16:19
  • 1
    $\begingroup$ @Björn Great catch. with the new corrected effect size tt_ind_solve_power(effect_size, power=1.0-beta, nobs1=None, alpha=alpha, alternative='two-sided') gives 44512.78 per group in statsmodels. $\endgroup$
    – Josef
    Commented May 30 at 14:03
1
$\begingroup$

Your problem statement has a few issues. First, the Lehr's rule is an approximate formula, not an exact one. But that would not explain such a large difference in estimated sample sizes. You state "I want to increase the conversion rate by 5%". This implies a binomial test (did web site visitor convert? Y/N). But you seem to want to use a t-test? I would suggest a 2x2 contingency matrix, with a Fisher-exact test. Yes, there is a normal approximation for this case, but its applicability depends on the expected proportion (it is not applicable if that proportion is close to 0, or 1).
To find the required sample size for a binomial test, one needs to know the expected proportion (you will get different sample sizes depending on whether the initial rate, you are trying to increase is 5%, 50%, or 90%, e.g.). From your code, it seems to be .1 (?).
When you say "increase the conversion rate by 5%", do you mean in absolute terms (e.g. going from 10% to 15%), or in relative terms (e.g. going from 10% to 10.5%). I am assuming it is most likely to be in absolute terms, as an absolute .05% change may not be commercially very significant (?). This seems to be what your code also implies (MDE=.05, not .005).
I also suspect that there is some confusion about the effect size (related to the above comments about absolute change vs. relative one). In your code, you feed $CVR*MDE=.005$ as the MDE (minimum detectable effect). That is very small compared to your $\sigma=.09$. Are you really trying to detect a change going fro 10%, to 10.5%?
Last, using Minitab to calculate this required sample size, assuming a starting proportionj of 10%, an effect of 5% in absolute terms (e.g. a desired proportion after intervention of 15%), a $\beta$ of .8 and $\alpha$ of .05, and a single sided test (you want to show your intervention increased the conversion rate: you are instead using a double sided test, but with $\alpha=.1$, so same thing), I get a sample size of 540 per group, so 1080 total. But if try to find a difference between 10% and 10.5%, I get 45,500 samples per group, i.e. 91,000 samples.

$\endgroup$
1
  • $\begingroup$ Thanks for your response! Indeed, I am interested in going from a conversion rate of 0.1 to 0.105. I am mostly concerned that the numbers for the sample size are orders of magnitude out, rather than they are 100% exact. I realise that there are better alternatives than a t-test, but the people I have to train have a low baseline of statistical knowledge and need something simple which is repeatable in software across several different metric types. $\endgroup$
    – Tom Kealy
    Commented May 28 at 14:34

Not the answer you're looking for? Browse other questions tagged or ask your own question.