I am running a study to test the effectiveness of treatment A compared to treatment B. Given the study limitations, I am only able to recruit 200 participants into treatment A. The estimated incidence rate of disease in the treatment group is expected to be 20%. What is the sample size required for treatment B if I want to observed at least 5% (0.05) difference in incidence rate between treatment A and B (assuming type 1 error 0.05, power 80%)?
1 Answer
If I understand the question correctly, it will be very difficult to distinguish between an incidence rate of 0.2 in Group A with only $n_A = 200$ subjects, and an incidence rate of 0.25 in Group B, regardless of the number $n_B$ of subjects in Group B.
The estimate of $\hat p_A = X/n_a = X/200,$ where $X \sim \mathsf{Binom}(200, 0.2),$ has standard error $SD(\hat p_A) \approx \sqrt{.2(.8)/200} \approx 0.028.$ Thus, the margin of error of a 95% confidence interval for $p_A$ will be about $\pm 1.96(0.028) \approx 0.055.$ However, you want to detect a difference of size $0.05$ between $p_A$ and $p_B,$ which seems difficult even if $p_B$ were known exactly.
In R, the procedure prop.test
is a test of two proportions.
If $n_A = n_B = 500,\,$ $X \sim \mathsf{Binom}(500, .2),$ and
$Y \sim \mathsf{Binom}(500, .25),$ then we might get $\hat p_A = 0.222,
\hat p_b = 0.236.$
set.seed(921)
x = rbinom(1, 500, .2); x
[1] 111
y = rbinom(1, 500, .25); y
[1] 118
x/500; y/500
[1] 0.222
[1] 0.236
Then prop.test
gives the following output, showing no significant
difference (P-value $0.60 > 0.05 = 5\%).$ [Continuity correction is
suppressed on account of sample sizes exceeding 100.]
prop.test(c(111,118), c(500,500), cor=F)
2-sample test for equality of proportions
without continuity correction
data: c(111, 118) out of c(500, 500)
X-squared = 0.27753, df = 1, p-value = 0.5983
alternative hypothesis: two.sided
95 percent confidence interval:
-0.06607901 0.03807901
sample estimates:
prop 1 prop 2
0.222 0.236
A simulation of many such cases with $n_A = n_b = 500$ shows that the null hypothesis is rejected in less than half of iterations. (That is, power less than 50%.)
By contrast, if $n_A = n_B = 1200,$ then one has power just above $80\%$ detecting a difference between $p_A = 0.20$ and $p_B = 0.25.$
set.seed(2020)
na = nb = 1200
pv=replicate(10^6,
prop.test(c(rbinom(1,na,.2),
rbinom(1,nb,.25)), c(na,nb))$p.val)
mean(pv <= .05)
[1] 0.823413