Calculating the optimal sample size before running a test

Question

I want to run an experiment to test the outcomes of variations A and B. I'm designing the experiment but I want to know what's the minimum sample size to obtain a result of 10% difference between treatment and control with 90% confidence and a power of at least 80%.

If the standard deviation of variations A and B are different, the formula for the sample size is

$$N = \left ( \frac{t_{0.05/2}+t_{1-0.8}}{\delta} \right)^2 \left ( \frac{\sigma_0^2}{\frac{\sigma_0}{\sigma_0 + \sigma_1}} + \frac{\sigma_1^2}{\frac{\sigma_1}{\sigma_0 + \sigma_1}} \right )$$

Where delta is the expected difference in the outcome, or 10% as mentioned above.

What we don't know is the standard deviations. We know $$\delta = p_1 - p_0 = 10\%$$ but we don't know before the experiment the value for each term. If we knew them we could calculate $$\sigma_0 = p_0(1-p_0)$$ $$\sigma_1 = p_1(1-p_1)$$

Is there a way to obtain an optimal sample size before running a test? Assume there is no literature available.

Answer 1

You are mixing sample sizes for confidence intervals with sample sizes for a given significance level and power against a particular alternative in a test of hypothesis.

Sample sizes for a pooled two-sided, two-sample t test. It is straightforward to find the sample size of each group for a pooled 2-sample t test, provided that you know (a) the size of difference you want to detect, (b) the significance level of your test (maybe 5% or 10% depending on the purpose of your investigation, (c) the desired power of the test, and the common standard standard deviation of the two populations.

The computation uses the noncentral t distribution. Most statistical software programs have 'power and sample size procedures' that do related computations. Here is output from a recent release of Minitab software. The parameters used are $\alpha = 0.05,$ power $0.8$ and $0.9,$ difference $\delta = 4,$ and standard deviation $\sigma = 10,$ for a two-sided test.

Power and Sample Size 

2-Sample t Test

Testing mean 1 = mean 2 (versus ≠)
Calculating power for mean 1 = mean 2 + difference
α = 0.05  Assumed standard deviation = 10


            Sample  Target
Difference    Size   Power  Actual Power
         4     100     0.8      0.803648
         4     133     0.9      0.901483

The sample size is for each group.

Simulated power of a Welch t test. Other tests may require simulation. Below I simulate the power (about $97\%)$ of a Welch 2-sample t test at 5% level, with standard deviations $\sigma_1 = 10, \sigma_2 = 15,$ sample sizes $n_1 = 100, n_2 = 150,$ and $\delta = 6.$

set.seed(2020)
pv = replicate(10^5, t.test(rnorm(100,100,10), 
                            rnorm(150,106,15))$p.val)
mean(pv < .05)
[1] 0.96664

[The Welch test does not assume equal variances. Without loss of generality I used $\mu_2 - \mu_1 = 106 - 100 = 6 = \delta.$ The power against $\mu_2 - \mu_1 = 26 - 20 = 6 = \delta$ would be the same, if other parameters are left unchanged.]