# Estimating minimal sample size for two sample test (comparing means)

If I want to perform a hypothesis test to compare the means of 2 equally big samples from normal distributions with equal variances, how would I compute the minimum required sample size for each sample if I wish to have a power, ($$1-\beta$$), of 0.8 to detect with $$\alpha = 0.05$$ a mean difference equal to half of the common standard deviation? Note that in this case we are testing $$H_0: \mu_1 - \mu_2 = \delta_0, H_A: \mu_1 - \mu_2 \neq \delta_0$$. And we have a Type II error if $$\mu_1 - \mu_2 = \delta$$.

So far this is what I have:

$$n \approx \frac{(z_{\alpha/2} + z_{1-\beta})^2(\sigma_1^2 + \sigma_2^2)}{(\delta-\delta_0)^2} = \dfrac{(1.96+0.8416)^2(2\sigma)}{(0.5\sigma - \delta_0)^2} = \dfrac{15.7\sigma}{(0.5\sigma - \delta_0)^2}$$

From this online calculator it seems like $$n = 63$$ but I am not sure why as I do not understand how to get the $$\delta_0$$ value? How would I use this as I am not given it?

If in fact $$\mu_1-\mu_2=\delta$$ then $$\bar X_1- \bar X_2 \sim N(\delta, \frac2n \sigma^2)$$

I would have thought you want $$\delta_0 =0$$ for your null hypothesis.

An $$\alpha=0.05$$ two-sided test would then reject the null hypothesis if $$\frac{\bar X_1- \bar X_2}{\sqrt{\frac2n}\sigma}< \Phi^{-1}(\frac{0.05}2)\approx -1.96$$ or $$\frac{\bar X_1- \bar X_2}{\sqrt{\frac2n}\sigma} > \Phi^{-1}(1-\frac{0.05}2)\approx 1.96$$

If in fact $$\delta = \frac12 \sigma$$ then the probability of this would be $$\Phi\left(\frac{-1.96\sqrt{\frac2n}\sigma -\frac12 \sigma}{\sqrt{\frac2n}\sigma}\right)+1-\Phi\left(\frac{1.96\sqrt{\frac2n}\sigma -\frac12 \sigma}{\sqrt{\frac2n}\sigma}\right)$$ which is $$\Phi\left({-1.96 -\sqrt{\frac n8}}\right)+\Phi\left({-1.96 +\sqrt{\frac n8}}\right)$$.

The first term is likely to turn out to be very small, so setting $$\Phi\left({-1.96 +\sqrt{\frac n2}}\right)\approx 0.8$$ suggests $$n \approx 8(\Phi^{-1}(0.8)+1.96)^2 \approx 8(0.84+1.96)^2 \approx 62.7$$ which rounds up to $$63$$

• Hey, thanks for the answer. Might be a silly question but how do you get $2\sigma^2/n$? And why isnt it 64 instead of 63 as shown by the other answer and the online calc? Both answers seem to make sense to me however. Apr 22, 2021 at 9:01
• @User_13 I see 63 in your online calculator ($mu1=0$, $mu2=0.5$, $sigma=1$, $2$-sided test, $\alpha=0.05$, $power=0.8$). Variance for $\bar{X}_i$ is $\frac1n \sigma^2$ so variance of their difference is twice that. BruceET's calculation uses a $t$-test initially and so has a slightly smaller critical region for rejection of $H_0$ leading to a slightly larger sample size needed for a given power Apr 22, 2021 at 9:13

Your analysis seems to assume that the common variance $$\sigma^2$$ is known and to use the normal distribution throughout. That works pretty well as long as the sample sizes are reasonably large. However, for smaller sample sizes--if the population variance is unknown--you'd need to use t tests and thus non-central t distributions for power computations. [For details, see advanced applied statistics tests, google something like pooled t power noncentral, or see this link.]

The effect size $$\Delta = \delta - \delta_0,$$ in your notation is important to power and sample size computation. (in applications, one often has $$\delta_0 = 0.)$$ You are correct it is $$\Delta/\sigma$$ that matters in the computation, where $$\sigma^2$$ is the common variance. You have stated that you want $$\Delta/\sigma = 1/2.$$

Here is Minitab output for 2-sided, pooled 2-sample t test at level $$\alpha = 0.05 = 5\%$$ with $$\sigma = 2, \Delta = 1.$$ [Sample sizes would be the same if I'd used $$\sigma = 10, \Delta = 5.]$$

Power and Sample Size

2-Sample t Test

Testing mean 1 = mean 2 (versus ≠)
Calculating power for mean 1 = mean 2 + difference
α = 0.05  Assumed standard deviation = 2

Sample  Target
Difference    Size   Power  Actual Power
1      64     0.8      0.801460

The sample size is for each group. However, if $$\sigma = \Delta,$$ then the the required sample size is is only $$n_1=n_2 = 17.$$ [Power plot omitted.]

Power and Sample Size

2-Sample t Test

Testing mean 1 = mean 2 (versus ≠)
Calculating power for mean 1 = mean 2 + difference
α = 0.05  Assumed standard deviation = 1

Sample  Target
Difference    Size   Power  Actual Power
1      17     0.8      0.807037

The sample size is for each group.


Many statistical software programs have power and sample size procedures for pooled 2-sample t tests. Because balanced designs with $$n_1=n_2$$ are most efficient, they mostly assume equal sample sizes. (There are also online power and sample size calculators online; they may vary in accuracy and ease of use.)

Also, you can easily simulate the power for given $$\alpha, \sigma, \Delta,$$ as shown below. [Technical details of R: The $-notation can be used to show just the P-value of a test; var.eq=T does a pooled test; the vector pv has 100,000 P-values; the logical vector pv <= .05 has 100,000 TRUEs and FALSEs and its mean is its ]proportion of TRUEs.] With 100,000 iterations, the power should be accurate to two places. The power agrees with Minitab's value, about 81%. set.seed(2021) pv = replicate(10^5, t.test(rnorm(17,10,2), rnorm(17,12,2), var.eq=T)$$p.val) mean(pv <= 0.05)  0.80728$$ simulated P-value  Finding the P-value for a Welch t test, which does not assume equal variances is not so straightforward. So power and sample size procedures for the Welch t test are not so widely available. However, simulation works about as easily. In the following we assume $$sigma_1 = 1.5, \sigma_2 = 2.5,$$ and find the power of a Welch 2-sample t test. The number of degrees of freedom is reduced for each individual test to allow for unequal sample variances, yielding slightly lower power: about 78%. set.seed(1234) pv = replicate(10^5, t.test(rnorm(17,10,1.5), rnorm(17,12,2.5))$p.val)
mean(pv <= 0.05)
 0.77676


If you really need to use an unbalanced design with $$n_1 \ne n_2$$ you can use simulation to approximate power for that as well.

• Thank you! But could you please tell my why the answer of 63 is incorrect as given by @Henry ? Because both your answers make sense? Apr 22, 2021 at 8:59
• My Answ starts with: "Your analysis seems to assume that the common variance $σ^2$ is known and to use the normal distribution throughout. That works pretty well as long as the sample sizes are reasonably large." I haven't checked every step of @Hentry's Answ, but it seems he too is assuming $\sigma$ known and finding sample size for 2-sample z test. With $n$ in the 60's I wouldn't be surprised if there is a little difference between z $(\sigma$ known) and t $(\sigma$ unknown) tests. Apr 22, 2021 at 15:30
• And would much change if the variance was not known? I guess your assumption is based on the fact that we know that both variances are equal which is fair. I guess either 63 or 64 would then be fine, but for robustness and to be on the "safer" side, you'd want 64? Apr 22, 2021 at 15:33
• My main point is that finding the necessary $n$ for a z test using the normal dist'n and the sample size for a t test found using a noncentral t dist'n won't necessarily be the same, even if you assume $\sigma=2.0$ in both computations. And of course results for both would change if you changed to $\sigma=2.1.$ // In practice, power and sample size computations are based on estimates of quantities that may be ill-defined (desired effect size) or unknowable (variance). So they are necessarily speculative. And if you plan for 64 subj one will get married, move to Idaho, and drop out, leaving 63. Apr 22, 2021 at 16:09