7
$\begingroup$

In a pilot study with two groups, the control distribution has the Mean1 = 90, SD1 = 5 and the treatment distribution has Mean2 = 85, SD2 = 5. The null hypothesis of the test is that the sample populations have the same mean. Find the effect size and determine the sample size needed for the test in which a power of 80% is acceptable, with the significance level at 5%.

My attempt: I calculated Cohen's $d = (\mu_1 - \mu_2) / \sigma = 1$ and used the following formula for the sample size of each group

$$n = \left( \frac{Z_{1-\alpha/2} + Z_{1-\beta}}{d} \right)^2$$

where $Z_x = \Phi^{-1}(x)$, obtaining $n = 7.84888$. I also tried to do this in python

from statsmodels.stats.power import NormalIndPower
obj = NormalIndPower()
obj.solve_power(effect_size=1, alpha=.05, power=.8, ratio=1.0)
>>> 15.6977

which is almost exactly twice my value. At first I thought this function returns the total number of samples for both groups but the documentation reads

solve for any one parameter of the power of a two sample z-test [...] nobs1 number of observations of sample 1. The number of observations of sample two is ratio times the size of sample 1

What am I missing? Is my formula correct?

$\endgroup$
5
  • $\begingroup$ The factor of 2 is due to one-sided vs two-sided test $\endgroup$
    – Ggjj11
    Feb 16 at 11:01
  • 1
    $\begingroup$ I think @Ggjj11 means a two sample vs one sample test. The formula is for a one-sample test (or a paired test) not test of two independent samples. $\endgroup$ Feb 16 at 11:15
  • $\begingroup$ @Ggjj11 solve_power is two-sided by default and the formula should also be two-sided since it uses $\alpha/2$ $\endgroup$
    – Christian
    Feb 16 at 11:21
  • $\begingroup$ @GeorgeSavva Do you know any good resources on that? $\endgroup$
    – Christian
    Feb 16 at 11:24
  • 1
    $\begingroup$ @Christian no specific resources, but look up the difference between one-sample and two-sample tests. I added a bit more detail as an answer below. You might also consider the difference between a z-test and a t-test. Also the questionable wisdom of using the Cohen's d from a pilot study as the basis for a power calculation for the next study. $\endgroup$ Feb 16 at 12:37

1 Answer 1

6
$\begingroup$

Your formula is for a one-sample test, that is for testing the null hypothesis that the mean of a population has a given value given a single sample from it.

You can check the 'one sample' calculation with the pwr package in R.

> pwr::pwr.norm.test(d=1, n=NULL, power=0.8)

     Mean power calculation for normal distribution with known variance 

              d = 1
              n = 7.84887
      sig.level = 0.05
          power = 0.8
    alternative = two.sided

I don't know of an R-package to do the two-sample Z test power calculation, probably because this is not something people very often want to do.

But intuitively, for a two sample Z-test, the difference in means will have a standard error that is $\sqrt{2}$ times the standard error for single group mean. So your sample size per group will need to be twice as large to achieve the same power, which agrees with the python result you quoted.

Usually though you'd use a t-test here not a Z test, because the standard deviation needs to be estimated from the data. Are you specifically instructed to use a Z-test?

For the t-test the sample size will be a little higher than the corresponding z-test.

> power.t.test(n=NULL, delta=5, sd=5, power=0.8)

     Two-sample t test power calculation 

              n = 16.71477
          delta = 5
             sd = 5
      sig.level = 0.05
          power = 0.8
    alternative = two.sided

NOTE: n is number in *each* group
```
$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.