1
$\begingroup$

when calculating the required sample size with a given power and p threshold, usually the calculator only requires two estimated population mean and standard deviation. is it always valid to assume the two populations' standard deviations are the same? what if they are not the same?

An example of sample size/power calculator here: https://www.stat.ubc.ca/~rollin/stats/ssize/n2.html

$\endgroup$
1
  • $\begingroup$ Your online calculator is for a pooled 2-sample t test, which assumes the two groups come from populations with the same variance. // Also, this calculator assumes $n_1=n_2.$ $\endgroup$
    – BruceET
    Commented Nov 24, 2020 at 23:00

1 Answer 1

2
$\begingroup$

For $\mu_1 = 100, \mu_2 = 110, \sigma = 10, \alpha = 0.05$ and desired power $0.8,$ your online calculator for a pooled 2-sample t test gave required sample sizes $n_1 = n_2 = 16.$

The following simulation in R can be used to see if this result is valid. With 100,000 iterations we can expect about two place accuracy, so agreement is pretty good.

set.seed(2020)
pv = replicate(10^5, 
 t.test(rnorm(16,100,10),rnorm(16,110,10),var.eq=T)$p.val)
mean(pv <= 0.05)
[1] 0.7812

If you want to know the power of a Welch t test (in which population SDs may differ) with parameters $\mu_1 = 100, \mu_2 = 110,$ $\sigma_1 = 10, \sigma_2=15,$ $n_1 = 12, n_2 = 20, \alpha = 0.05,$ then you can make obvious changes in the simulation to obtain power about 59%.

set.seed(1124)
pv = replicate(10^5, 
 t.test(rnorm(12,100,10),rnorm(20,110,15))$p.val)
mean(pv <= 0.05)
[1] 0.58547

So, in order to accommodate the different SDs (one larger than above), you might consider increasing sample sizes to $n_1 = 25, n_2=35,$ assuming it is easier or cheaper to sample from the second population. Then the power is about 86%.

set.seed(1234)
pv = replicate(10^5, 
 t.test(rnorm(25,100,10),rnorm(35,110,15))$p.val)
mean(pv <= 0.05)
[1]  0.85936

Roughly speaking, balanced designs with $n_1=n_2,$ give the best power for a given total sample size $n_1+n_2.$ Unless there is a good reason for using an unbalanced design, it is best to avoid them.

Notes: For pooled tests, there are fairly simple formulas involving noncentral t distributions, which make simulation unnecessary. If population SDs do not differ by much, you might use formulas for a pooled test as a starting place for designing an experiment for a Welch test. I have not seen formulas for Welch tests; only simulation results.

$\endgroup$
2
  • 1
    $\begingroup$ (+1) Re. "power of a Welch t test" (3rd §): This should probably read t.test(..., var.eq=F). The following discusses a closed-form solution for the case of unequal variances: Schouten, H. J. A. (1999). Sample size formula with a continuous outcome for unequal group sizes and unequal variances. Statistics in Medicine, 18, 87–91. $\endgroup$
    – chl
    Commented Nov 25, 2020 at 16:54
  • $\begingroup$ OK, I see what you mean. Mistake corrected. Thanks much. $\endgroup$
    – BruceET
    Commented Nov 25, 2020 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.