Estimating minimal sample size for two sample test (comparing means)

Question

If I want to perform a hypothesis test to compare the means of 2 equally big samples from normal distributions with equal variances, how would I compute the minimum required sample size for each sample if I wish to have a power, ($1-\beta$), of 0.8 to detect with $\alpha = 0.05$ a mean difference equal to half of the common standard deviation? Note that in this case we are testing $H_0: \mu_1 - \mu_2 = \delta_0, H_A: \mu_1 - \mu_2 \neq \delta_0$. And we have a Type II error if $\mu_1 - \mu_2 = \delta$.

So far this is what I have:

$$n \approx \frac{(z_{\alpha/2} + z_{1-\beta})^2(\sigma_1^2 + \sigma_2^2)}{(\delta-\delta_0)^2} = \dfrac{(1.96+0.8416)^2(2\sigma)}{(0.5\sigma - \delta_0)^2} = \dfrac{15.7\sigma}{(0.5\sigma - \delta_0)^2}$$

From this online calculator it seems like $n = 63$ but I am not sure why as I do not understand how to get the $\delta_0$ value? How would I use this as I am not given it?

Answer 1

If in fact $\mu_1-\mu_2=\delta$ then $\bar X_1- \bar X_2 \sim N(\delta, \frac2n \sigma^2)$

I would have thought you want $\delta_0 =0$ for your null hypothesis.

An $\alpha=0.05$ two-sided test would then reject the null hypothesis if $\frac{\bar X_1- \bar X_2}{\sqrt{\frac2n}\sigma}< \Phi^{-1}(\frac{0.05}2)\approx -1.96$ or $\frac{\bar X_1- \bar X_2}{\sqrt{\frac2n}\sigma} > \Phi^{-1}(1-\frac{0.05}2)\approx 1.96$

If in fact $\delta = \frac12 \sigma$ then the probability of this would be $\Phi\left(\frac{-1.96\sqrt{\frac2n}\sigma -\frac12 \sigma}{\sqrt{\frac2n}\sigma}\right)+1-\Phi\left(\frac{1.96\sqrt{\frac2n}\sigma -\frac12 \sigma}{\sqrt{\frac2n}\sigma}\right)$ which is $\Phi\left({-1.96 -\sqrt{\frac n8}}\right)+\Phi\left({-1.96 +\sqrt{\frac n8}}\right)$.

The first term is likely to turn out to be very small, so setting $\Phi\left({-1.96 +\sqrt{\frac n2}}\right)\approx 0.8$ suggests $ n \approx 8(\Phi^{-1}(0.8)+1.96)^2 \approx 8(0.84+1.96)^2 \approx 62.7$ which rounds up to $63$

Answer 2

Your analysis seems to assume that the common variance $\sigma^2$ is known and to use the normal distribution throughout. That works pretty well as long as the sample sizes are reasonably large. However, for smaller sample sizes--if the population variance is unknown--you'd need to use t tests and thus non-central t distributions for power computations. [For details, see advanced applied statistics tests, google something like pooled t power noncentral, or see this link.]

The effect size $\Delta = \delta - \delta_0,$ in your notation is important to power and sample size computation. (in applications, one often has $\delta_0 = 0.)$ You are correct it is $\Delta/\sigma$ that matters in the computation, where $\sigma^2$ is the common variance. You have stated that you want $\Delta/\sigma = 1/2.$

Here is Minitab output for 2-sided, pooled 2-sample t test at level $\alpha = 0.05 = 5\%$ with $\sigma = 2, \Delta = 1.$ [Sample sizes would be the same if I'd used $\sigma = 10, \Delta = 5.]$

Power and Sample Size 

2-Sample t Test

Testing mean 1 = mean 2 (versus ≠)
Calculating power for mean 1 = mean 2 + difference
α = 0.05  Assumed standard deviation = 2

             Sample  Target
Difference    Size   Power  Actual Power
         1      64     0.8      0.801460

The sample size is for each group.

However, if $\sigma = \Delta,$ then the the required sample size is is only $n_1=n_2 = 17.$ [Power plot omitted.]

Power and Sample Size 

2-Sample t Test

Testing mean 1 = mean 2 (versus ≠)
Calculating power for mean 1 = mean 2 + difference
α = 0.05  Assumed standard deviation = 1

            Sample  Target
Difference    Size   Power  Actual Power
         1      17     0.8      0.807037

The sample size is for each group.

Many statistical software programs have power and sample size procedures for pooled 2-sample t tests. Because balanced designs with $n_1=n_2$ are most efficient, they mostly assume equal sample sizes. (There are also online power and sample size calculators online; they may vary in accuracy and ease of use.)

Also, you can easily simulate the power for given $\alpha, \sigma, \Delta,$ as shown below. [Technical details of R: The $-notation can be used to show just the P-value of a test; var.eq=T does a pooled test; the vector pv has 100,000 P-values; the logical vector pv <= .05 has 100,000 TRUEs and FALSEs and its mean is its ]proportion of TRUEs.] With 100,000 iterations, the power should be accurate to two places. The power agrees with Minitab's value, about 81%.

set.seed(2021)
pv = replicate(10^5, t.test(rnorm(17,10,2), 
                            rnorm(17,12,2), var.eq=T)$p.val)
mean(pv <= 0.05)
[1] 0.80728     $ simulated P-value

Finding the P-value for a Welch t test, which does not assume equal variances is not so straightforward. So power and sample size procedures for the Welch t test are not so widely available. However, simulation works about as easily. In the following we assume $sigma_1 = 1.5, \sigma_2 = 2.5,$ and find the power of a Welch 2-sample t test. The number of degrees of freedom is reduced for each individual test to allow for unequal sample variances, yielding slightly lower power: about 78%.

set.seed(1234)
pv = replicate(10^5, t.test(rnorm(17,10,1.5), 
                            rnorm(17,12,2.5))$p.val)
mean(pv <= 0.05)
[1] 0.77676

If you really need to use an unbalanced design with $n_1 \ne n_2$ you can use simulation to approximate power for that as well.