3
$\begingroup$

I have two groups of very unevenly distributed data, one group has 95% more observations than the other group. My outcome variable is a binary variable.

I know how to calculate the effect size, but I'm curious how to go from there to finding out the sample size for a certain power and significance. I know the formulas, I just don't have an understanding of why.

$Cohen's \ d$ = $ES$ = $\frac{\mu_1 - \mu_2}{s_p}$ (see here),

and the formula for determining the sample sizes to ensure that the test has a specified power is given below (see here)

$n_i$ = $2*(\frac{Z_{1-\alpha/2}+Z_{1-\beta}}{ES})^2$

Can anybody explain the idea/motivation behind this last formula?

Improve this question
$\endgroup$
8
  • $\begingroup$ I find a similar question here: stats.stackexchange.com/questions/478288/…, but no discussion as to the formula $\endgroup$
    – Berthrand Eros
    Oct 6, 2021 at 14:46
  • 1
    $\begingroup$ You mentioned the data was binary but you applied the effect size formula for difference in means... $\endgroup$
    – Pitouille
    Oct 6, 2021 at 17:31
  • $\begingroup$ @Pitouille: Yes, I want to find out if the mean of the outcomes are different across the two groups. I see that the group with the low number of observations has a higher mean, but since the number of observations are so low and the uncertainty is so high, I need to draw more samples - hence the need for using effect size and sample sizes. I didn't want to explain too much, as I'm mostly interested in the theoretical and motivational aspect behind the sample size formula, but I apologize if I just made things confusing $\endgroup$
    – Berthrand Eros
    Oct 6, 2021 at 17:56
  • $\begingroup$ The link is long, somewhat diffuse, not as careful about notation as I would like, not always clear whether $n$ is for tests or confidence intervals, and considers various cases for continuous and categorical data. I think you may have taken $ES$ from one section and $n$ from another. Perhaps Example 6 illustrates the situation you describe in your Question.. $\endgroup$
    – BruceET
    Oct 6, 2021 at 18:42
  • 2
    $\begingroup$ If you want to understand the idea behind the formula, I think you can get useful information from the wikipedia page: en.wikipedia.org/wiki/Sample_size_determination $\endgroup$
    – Pitouille
    Oct 7, 2021 at 7:27

2 Answers 2

Reset to default
5
$\begingroup$

I suppose you are considering a study with sample sizes large enough that it is appropriate to use an approximate normal test.

Suppose you want to compare two populations $(1$ and $2)$ with Success probabilities $p_1, p_2$ and sample sizes $n_1, n_2,$ respectively and that you want to be reasonably sure to reject $H_0: p_1 - p_2 = 0$ at the 5% level against a two-sided alternative, if $|p_1 - p_2| > .04.$

Suppose you have data as below:

set.seed(1234)
n1 = n2 = 3500
p1 = .60;  p2 = .64
x1 = rbinom(1, n1, p1);  x1
[1] 2128  
x2 = rbinom(1, n2, p2);  x2
[1] 2227   
prop.test(c(x1,x2), c(n1,n2), cor=F)$p.val
[1] 0.01466729

So sample sizes of 4000 were sufficient to reject the null hypothesis in this particular case. The full output (not just the P-value) is shown below. I declined continuity correction on account of the large sample sizes.

prop.test(c(x1,x2), c(n1,n2), cor=F)

        2-sample test for equality of proportions 
        without continuity correction

data:  c(x1, x2) out of c(n1, n2)
X-squared = 5.956, df = 1, p-value = 0.01467
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.050992366 -0.005579062
sample estimates:
   prop 1    prop 2 
0.6080000 0.6362857

Now the question is whether this particular dataset is typical or just lucky by chance. We can do a simulation to approximate the power (probability $H_0$ is rejected) in such experiments.

set.seed(2021)
n1 = n2 = 3500;  p1 = .6;  p2 = .64
pv=replicate(10^4, prop.test( c(rbinom(1,n1,p1),rbinom(1,n2,p2)), 
              c(n1,n2),cor=F)$p.val)
mean(pv <= .05)
[1] 0.9324

The power of prop.test for the specified parameters is about $93\%,$ So with samples of size 3500 you have a good chance that the test will find a real difference between success probabilities 0.60 and 0.64. [Note: Differences of 0.04 between $p_1$ and $p_2$ are somewhat easier to detect near 0 and 1 (say 0.10 vs. 0.14) and harder to detect near 0.5 (say 0.48 vs. 0.52). Simulation takes this into account, an approximate formula for sample size may or may not do so.]

set.seed(2021)
n1 = n2 = 3500;  p1 = .1;  p2 = .14
pv=replicate(10^4, prop.test( c(rbinom(1,n1,p1),rbinom(1,n2,p2)), 
              c(n1,n2),cor=F)$p.val)
mean(pv <= .05)
[1] 0.9992


set.seed(2021)
n1 = n2 = 3500;  p1 = .48;  p2 = .52
pv=replicate(10^4, prop.test( c(rbinom(1,n1,p1),rbinom(1,n2,p2)), 
              c(n1,n2),cor=F)$p.val)
mean(pv <= .05)
[1] 0.9137

When you have a useful formula for computing sample size, you can plug in the values I used above to see if you get about the same answer I got by simulation. Alternatively, you can forget about finding a formula for $n$ and change the parameters in my simulation to find the sample size that matches your situation.

There are some online 'power and sample size' calculators on the Internet. If you want to explore this, try to find one sponsored by a government agency or the statistics department of a major university.

Improve this answer
$\endgroup$
2
  • $\begingroup$ This is a really great answer, thank you very much! I don't seem to able to upvote yet, but hope I can do that soon! I found a power and sample size calculator online yesterday from a statistics dpt, but it bothered me that I didn't know how the calculations were being done. And it also bothers me with the formula above (from example 11 in my second link), that I don't understand the motivation behind the formula. Some guy didn't just make this formula out of nothing, if you catch my thinking. Anyway, your approach gives me a brilliant way of testing sample sizes on my own, I'm so grateful! $\endgroup$
    – Berthrand Eros
    Oct 7, 2021 at 6:02
  • 1
    $\begingroup$ Glad my answer was helpful. No worries about upvote. // But FYI, I think you can click the check mark to Accept from the start, and you should soon have enough 'reputation' points to Up-vote whatever you like. $\endgroup$
    – BruceET
    Oct 7, 2021 at 16:35
2
$\begingroup$

To complement the existing answer with a more visual/intuitive aspect of the sample size formula you mentioned. As you might have read from the Wikipedia page that these formulas actually origin from the type of test to want to perform.

Here you can find a simple/intuitive explanation of the numerator (where power is involved: Power Analysis Made Easy. I reproduced the same diagram keeping the Z value for ease of understanding:

enter image description here

Of course, the (pooled) standard deviation does not appear in the equation because it is hidden within ES, the effect size.

Finally, to complete the intuitive aspect of this answer, here is a nice tool that allows you to play with the different parameters involved in the formula: https://rpsychologist.com/d3/nhst/

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Thank you very much, I'm very grateful for such amazing help! The visual is really great as well! $\endgroup$
    – Berthrand Eros
    Oct 8, 2021 at 8:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.