In your situation, with very small values of $p,$ you
need to be careful about using normal approximations--either for testing or for determining sample size to
achieve a desired power.
Suppose your null hypothesis is $H_0: p=.01$ and $H_a: p > .01$ and you want
power .8 or .9 against the specific alternative
$p_a = 0.04.$ Below are results from a release of Minitab
statistical software from a few years ago, which uses a normal approximation. According to this, you will need $n = 120$ (for 80% power) or $n = 192$ (for 90%).
Power and Sample Size
Test for One Proportion
Testing p = 0.01 (versus > 0.01)
α = 0.05
Sample Target
Comparison p Size Power Actual Power
0.04 120 0.8 0.800070
0.04 192 0.9 0.900803
There are several ways to test $H_0$ against $H_a,$
(a) using a normal approximation (with or without a continuity correction) and (b) using exact binomial CDFs.
For $p_a$ as small as $0.04,$ normal
approximations may not be valid. So it is best to
use an exact binomial test. In R, the exact binom.test for $x = 6$ cancer cases in $n=200$ rejects $H_0:p=0.01$ in favor of $H_a: p > 0.01$ with P-value $0.016 < 0.05 = 5\%.$
binom.test(6, 200, p=.01, alt="g")
Exact binomial test
data: 6 and 200
number of successes = 6, number of trials = 200, p-
value = 0.01602
alternative hypothesis:
true probability of success is greater than 0.01
95 percent confidence interval:
0.01314399 1.00000000
sample estimates:
probability of success
0.03
If we do 100,000 such tests with $X \sim\mathsf{Binom}(200, .04),$ then we might expect about 90% rejections according to the Minitab output
indicating a power of about 90%.
However, using the exact binomial test in the simulation below, we get power only about 82%.
set.seed(2021)
pv = replicate(10^5, binom.test(rbinom(1,200,.04), 200, p=0.01, alt="gr")$p.val)
mean(pv < 0.05)
[1] 0.81542
By contrast, if we use prop.test in a similar simulation, we get power about 90%.
set.seed(2021)
pv = replicate(10^5, prop.test(rbinom(1,200,.04), 200, p=0.01, alt="gr")$p.val)
mean(pv < 0.05)
[1] 0.90554
The difficulty
is that many of the 100,000 tests in the second simulation used $X$-values sufficiently small to generate 'warning' messages that P-values might
not be accurate.
For example, if $X = 5,$ then prop.test gives a P-value below 5%, leading to rejection, but with a warning message that should not be ignored.
prop.test(5, 200, .01, alt="gr")$p.val
[1] 0.03781106
Warning message:
In prop.test(5, 200, 0.01, alt = "gr") :
Chi-squared approximation may be incorrect
By contrast, binom.test correctly gives a P-value above 5%, and so fails to reject: $P(X \ge 5|H_0) = 0.0517 > 0.05.$
binom.test(5, 200, .01, alt="gr")$p.val
[1] 0.05174626
1 - pbinom(4, 200, .01)
[1] 0.05174626
Unfortunately, not all software and online
'power and sample size' procedures reveal clearly what
assumptions underlie their computations.
Here are plots of the PDF of $\mathsf{Binom}(n=200,p=.04)$ and $\mathsf{Binom}(n=200,p=.01)$ along with the density
curves (orange) of the normal distributions with matching means and standard deviations.
x = 0:20; PDF = dbinom(x, 200, .04)
hdr = "PDF of BINOM(200, .04) with Density of NORM(8, 2.098)"
plot(x, PDF, type="h", col="blue", lwd=2, main=hdr)
curve(dnorm(x,mu,sg), add=T, col="orange", lwd=2)
abline(h=0, col="green2")
abline(v=0, col="green2")
