# Sample sizes for two independent samples w dichotomous outcomes

I have two groups of very unevenly distributed data, one group has 95% more observations than the other group. My outcome variable is a binary variable.

I know how to calculate the effect size, but I'm curious how to go from there to finding out the sample size for a certain power and significance. I know the formulas, I just don't have an understanding of why.

$$Cohen's \ d$$ = $$ES$$ = $$\frac{\mu_1 - \mu_2}{s_p}$$ (see here),

and the formula for determining the sample sizes to ensure that the test has a specified power is given below (see here)

$$n_i$$ = $$2*(\frac{Z_{1-\alpha/2}+Z_{1-\beta}}{ES})^2$$

Can anybody explain the idea/motivation behind this last formula?

• I find a similar question here: stats.stackexchange.com/questions/478288/…, but no discussion as to the formula Oct 6, 2021 at 14:46
• You mentioned the data was binary but you applied the effect size formula for difference in means... Oct 6, 2021 at 17:31
• @Pitouille: Yes, I want to find out if the mean of the outcomes are different across the two groups. I see that the group with the low number of observations has a higher mean, but since the number of observations are so low and the uncertainty is so high, I need to draw more samples - hence the need for using effect size and sample sizes. I didn't want to explain too much, as I'm mostly interested in the theoretical and motivational aspect behind the sample size formula, but I apologize if I just made things confusing Oct 6, 2021 at 17:56
• The link is long, somewhat diffuse, not as careful about notation as I would like, not always clear whether $n$ is for tests or confidence intervals, and considers various cases for continuous and categorical data. I think you may have taken $ES$ from one section and $n$ from another. Perhaps Example 6 illustrates the situation you describe in your Question.. Oct 6, 2021 at 18:42
• If you want to understand the idea behind the formula, I think you can get useful information from the wikipedia page: en.wikipedia.org/wiki/Sample_size_determination Oct 7, 2021 at 7:27

I suppose you are considering a study with sample sizes large enough that it is appropriate to use an approximate normal test.

Suppose you want to compare two populations $$(1$$ and $$2)$$ with Success probabilities $$p_1, p_2$$ and sample sizes $$n_1, n_2,$$ respectively and that you want to be reasonably sure to reject $$H_0: p_1 - p_2 = 0$$ at the 5% level against a two-sided alternative, if $$|p_1 - p_2| > .04.$$

Suppose you have data as below:

set.seed(1234)
n1 = n2 = 3500
p1 = .60;  p2 = .64
x1 = rbinom(1, n1, p1);  x1
[1] 2128
x2 = rbinom(1, n2, p2);  x2
[1] 2227
prop.test(c(x1,x2), c(n1,n2), cor=F)$p.val [1] 0.01466729  So sample sizes of 4000 were sufficient to reject the null hypothesis in this particular case. The full output (not just the P-value) is shown below. I declined continuity correction on account of the large sample sizes. prop.test(c(x1,x2), c(n1,n2), cor=F) 2-sample test for equality of proportions without continuity correction data: c(x1, x2) out of c(n1, n2) X-squared = 5.956, df = 1, p-value = 0.01467 alternative hypothesis: two.sided 95 percent confidence interval: -0.050992366 -0.005579062 sample estimates: prop 1 prop 2 0.6080000 0.6362857  Now the question is whether this particular dataset is typical or just lucky by chance. We can do a simulation to approximate the power (probability $$H_0$$ is rejected) in such experiments. set.seed(2021) n1 = n2 = 3500; p1 = .6; p2 = .64 pv=replicate(10^4, prop.test( c(rbinom(1,n1,p1),rbinom(1,n2,p2)), c(n1,n2),cor=F)$p.val)
mean(pv <= .05)
[1] 0.9324


The power of prop.test for the specified parameters is about $$93\%,$$ So with samples of size 3500 you have a good chance that the test will find a real difference between success probabilities 0.60 and 0.64. [Note: Differences of 0.04 between $$p_1$$ and $$p_2$$ are somewhat easier to detect near 0 and 1 (say 0.10 vs. 0.14) and harder to detect near 0.5 (say 0.48 vs. 0.52). Simulation takes this into account, an approximate formula for sample size may or may not do so.]

set.seed(2021)
n1 = n2 = 3500;  p1 = .1;  p2 = .14
pv=replicate(10^4, prop.test( c(rbinom(1,n1,p1),rbinom(1,n2,p2)),
c(n1,n2),cor=F)$p.val) mean(pv <= .05) [1] 0.9992 set.seed(2021) n1 = n2 = 3500; p1 = .48; p2 = .52 pv=replicate(10^4, prop.test( c(rbinom(1,n1,p1),rbinom(1,n2,p2)), c(n1,n2),cor=F)$p.val)
mean(pv <= .05)
[1] 0.9137


When you have a useful formula for computing sample size, you can plug in the values I used above to see if you get about the same answer I got by simulation. Alternatively, you can forget about finding a formula for $$n$$ and change the parameters in my simulation to find the sample size that matches your situation.

There are some online 'power and sample size' calculators on the Internet. If you want to explore this, try to find one sponsored by a government agency or the statistics department of a major university.

• This is a really great answer, thank you very much! I don't seem to able to upvote yet, but hope I can do that soon! I found a power and sample size calculator online yesterday from a statistics dpt, but it bothered me that I didn't know how the calculations were being done. And it also bothers me with the formula above (from example 11 in my second link), that I don't understand the motivation behind the formula. Some guy didn't just make this formula out of nothing, if you catch my thinking. Anyway, your approach gives me a brilliant way of testing sample sizes on my own, I'm so grateful! Oct 7, 2021 at 6:02
• Glad my answer was helpful. No worries about upvote. // But FYI, I think you can click the check mark to Accept from the start, and you should soon have enough 'reputation' points to Up-vote whatever you like. Oct 7, 2021 at 16:35

To complement the existing answer with a more visual/intuitive aspect of the sample size formula you mentioned. As you might have read from the Wikipedia page that these formulas actually origin from the type of test to want to perform.

Here you can find a simple/intuitive explanation of the numerator (where power is involved: Power Analysis Made Easy. I reproduced the same diagram keeping the Z value for ease of understanding:

Of course, the (pooled) standard deviation does not appear in the equation because it is hidden within ES, the effect size.

Finally, to complete the intuitive aspect of this answer, here is a nice tool that allows you to play with the different parameters involved in the formula: https://rpsychologist.com/d3/nhst/

• Thank you very much, I'm very grateful for such amazing help! The visual is really great as well! Oct 8, 2021 at 8:29