I want to run an experiment to test the outcomes of variations A and B. I'm designing the experiment but I want to know what's the minimum sample size to obtain a result of 10% difference between treatment and control with 90% confidence and a power of at least 80%.

If the standard deviation of variations A and B are different, the formula for the sample size is

$$N = \left ( \frac{t_{0.05/2}+t_{1-0.8}}{\delta} \right)^2 \left ( \frac{\sigma_0^2}{\frac{\sigma_0}{\sigma_0 + \sigma_1}} + \frac{\sigma_1^2}{\frac{\sigma_1}{\sigma_0 + \sigma_1}} \right )$$

Where delta is the expected difference in the outcome, or 10% as mentioned above.

What we don't know is the standard deviations. We know $$\delta = p_1 - p_0 = 10\%$$ but we don't know before the experiment the value for each term. If we knew them we could calculate $$\sigma_0 = p_0(1-p_0)$$ $$\sigma_1 = p_1(1-p_1)$$

Is there a way to obtain an optimal sample size before running a test? Assume there is no literature available.

  • $\begingroup$ Power and sample size computations for t test work better if you specify an absolute difference rather than a percentage difference. And you need an estimate of the standard deviation. $\endgroup$ – BruceET May 21 '20 at 5:45

You are mixing sample sizes for confidence intervals with sample sizes for a given significance level and power against a particular alternative in a test of hypothesis.

Sample sizes for a pooled two-sided, two-sample t test. It is straightforward to find the sample size of each group for a pooled 2-sample t test, provided that you know (a) the size of difference you want to detect, (b) the significance level of your test (maybe 5% or 10% depending on the purpose of your investigation, (c) the desired power of the test, and the common standard standard deviation of the two populations.

The computation uses the noncentral t distribution. Most statistical software programs have 'power and sample size procedures' that do related computations. Here is output from a recent release of Minitab software. The parameters used are $\alpha = 0.05,$ power $0.8$ and $0.9,$ difference $\delta = 4,$ and standard deviation $\sigma = 10,$ for a two-sided test.

Power and Sample Size 

2-Sample t Test

Testing mean 1 = mean 2 (versus ≠)
Calculating power for mean 1 = mean 2 + difference
α = 0.05  Assumed standard deviation = 10

            Sample  Target
Difference    Size   Power  Actual Power
         4     100     0.8      0.803648
         4     133     0.9      0.901483

The sample size is for each group.

enter image description here

Simulated power of a Welch t test. Other tests may require simulation. Below I simulate the power (about $97\%)$ of a Welch 2-sample t test at 5% level, with standard deviations $\sigma_1 = 10, \sigma_2 = 15,$ sample sizes $n_1 = 100, n_2 = 150,$ and $\delta = 6.$

pv = replicate(10^5, t.test(rnorm(100,100,10), 
mean(pv < .05)
[1] 0.96664

[The Welch test does not assume equal variances. Without loss of generality I used $\mu_2 - \mu_1 = 106 - 100 = 6 = \delta.$ The power against $\mu_2 - \mu_1 = 26 - 20 = 6 = \delta$ would be the same, if other parameters are left unchanged.]

  • $\begingroup$ This doesn't answer the question though. You are assuming you know the standard deviations and the individual mean, which is exactly the point of the original question. $\endgroup$ – NicolasVega May 22 '20 at 20:34
  • $\begingroup$ @NicolasVega: Granted. My comment says (or gently hints anyhow) that the orig'l question is not answerable b/c difference is expressed as %, not actual size and SDs not specified. My Answer tries to show a feasible scenario in which sample size can be determined, hoping you are able to think more deeply/realistically about your actual situation. You are certainly not the first experimenter to experience this kind of difficulty. Getting researchers to speculate on approx size of SDs can be the hardest part of consulting about experimental design in general and sample size in particular. $\endgroup$ – BruceET May 22 '20 at 21:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.