2
$\begingroup$

I'm having a disconnect on power and sample size estimation. I'm using the sample size calculator here: https://select-statistics.co.uk/calculators/sample-size-calculator-population-proportion/

Im trying to estimate how much sample I need to estimate a population with a small proportion. Say the population proportion for cancer is 1% and I want to be 95% confident that my error is 3%. I would imagine that I would need a large sample size if the population proportion is 1%, but the calculator gives me n=43. I don't understand how I could reasonably estimate such a small proportion with n=43. If only 1 out of every 100 folks in the population have cancer, how could I detect it with a sample of 43 people?

$\endgroup$
5
  • $\begingroup$ Do you intend for the 3% error to be relative to 1% (i.e. 0.97% - 1.03%) or absolute to 1% (i.e. "-2%" to 4%)? $\endgroup$
    – B.Liu
    May 18, 2021 at 15:30
  • $\begingroup$ Hey B.Liu, I would expect it to be absolute here $\endgroup$
    – Daniel
    May 18, 2021 at 15:34
  • $\begingroup$ 3% error for 1% proportion is very big... the chances that it will be ~0% is very high and the chances that you have more than 4% is very low.. $\endgroup$
    – ofer-a
    May 18, 2021 at 15:41
  • $\begingroup$ Right, but what sample size would I need to estimate such a proportion from a random sample? $\endgroup$
    – Daniel
    May 18, 2021 at 16:26
  • $\begingroup$ Notice that your link is a procedure for determining $n$ for confidence intervals, not power for tests. Related but not the same. If the link refers to Wald CIs (as is standard), then you have to remember those have poor coverage probability for small $n$ and for $p$ far from $1/2.$ {Maybe look at extensive Wikipedia material on binomial CIs.} $\endgroup$
    – BruceET
    May 18, 2021 at 23:50

3 Answers 3

2
$\begingroup$

In your situation, with very small values of $p,$ you need to be careful about using normal approximations--either for testing or for determining sample size to achieve a desired power.

Suppose your null hypothesis is $H_0: p=.01$ and $H_a: p > .01$ and you want power .8 or .9 against the specific alternative $p_a = 0.04.$ Below are results from a release of Minitab statistical software from a few years ago, which uses a normal approximation. According to this, you will need $n = 120$ (for 80% power) or $n = 192$ (for 90%).

Power and Sample Size 

Test for One Proportion

Testing p = 0.01 (versus > 0.01)
α = 0.05

              Sample  Target
Comparison p    Size   Power  Actual Power
        0.04     120     0.8      0.800070
        0.04     192     0.9      0.900803

There are several ways to test $H_0$ against $H_a,$ (a) using a normal approximation (with or without a continuity correction) and (b) using exact binomial CDFs.

For $p_a$ as small as $0.04,$ normal approximations may not be valid. So it is best to use an exact binomial test. In R, the exact binom.test for $x = 6$ cancer cases in $n=200$ rejects $H_0:p=0.01$ in favor of $H_a: p > 0.01$ with P-value $0.016 < 0.05 = 5\%.$

binom.test(6, 200, p=.01, alt="g")

        Exact binomial test

data:  6 and 200
number of successes = 6, number of trials = 200, p- 
value = 0.01602
alternative hypothesis: 
  true probability of success is greater than 0.01
95 percent confidence interval:
 0.01314399 1.00000000
sample estimates:
probability of success 
                  0.03 

If we do 100,000 such tests with $X \sim\mathsf{Binom}(200, .04),$ then we might expect about 90% rejections according to the Minitab output indicating a power of about 90%.

However, using the exact binomial test in the simulation below, we get power only about 82%.

set.seed(2021)
pv = replicate(10^5, binom.test(rbinom(1,200,.04), 200, p=0.01, alt="gr")$p.val)
mean(pv < 0.05)
[1] 0.81542

By contrast, if we use prop.test in a similar simulation, we get power about 90%.

set.seed(2021)
pv = replicate(10^5, prop.test(rbinom(1,200,.04), 200, p=0.01, alt="gr")$p.val)
mean(pv < 0.05)
[1] 0.90554

The difficulty is that many of the 100,000 tests in the second simulation used $X$-values sufficiently small to generate 'warning' messages that P-values might not be accurate.

For example, if $X = 5,$ then prop.test gives a P-value below 5%, leading to rejection, but with a warning message that should not be ignored.

prop.test(5, 200, .01, alt="gr")$p.val
[1] 0.03781106
Warning message:
In prop.test(5, 200, 0.01, alt = "gr") :
  Chi-squared approximation may be incorrect

By contrast, binom.test correctly gives a P-value above 5%, and so fails to reject: $P(X \ge 5|H_0) = 0.0517 > 0.05.$

binom.test(5, 200, .01, alt="gr")$p.val
[1] 0.05174626
1 - pbinom(4, 200, .01)
[1] 0.05174626

Unfortunately, not all software and online 'power and sample size' procedures reveal clearly what assumptions underlie their computations.

Here are plots of the PDF of $\mathsf{Binom}(n=200,p=.04)$ and $\mathsf{Binom}(n=200,p=.01)$ along with the density curves (orange) of the normal distributions with matching means and standard deviations.

x = 0:20;  PDF = dbinom(x, 200, .04)
hdr = "PDF of BINOM(200, .04) with Density of NORM(8, 2.098)"
plot(x, PDF, type="h", col="blue", lwd=2, main=hdr)
 curve(dnorm(x,mu,sg), add=T, col="orange", lwd=2)
 abline(h=0, col="green2")
 abline(v=0, col="green2")

enter image description here

enter image description here

$\endgroup$
2
$\begingroup$

A few thoughts beyond what BruceET provided in an extensive answer (+1).

First, the calculator page you link itself warns with respect to their method:

If, the sample proportion is close to 0 or 1 then this approximation is not valid and you need to consider an alternative sample size calculation method.

A reason is that the normal-approximation formula used on that page assumes symmetry in potential results around the estimated prevalence. If you are allowing for +/- 3 percentage points error around an estimated 1% population prevalence, the lower limit of 0 in probability means that you've destroyed that symmetry; you can't get a -2% probability.

If you use that same calculator and instead ask for +/- 0.99 percentage point error around the 1% estimated prevalence (that is, allowing a range between 0.01% and 1.99%), you get an estimated sample size of 387. That's probably more in line with what you were expecting to find.

Second, there is a large number of ways to estimate binomial proportion confidence intervals. The R binom package provides 11 of them. As the results of binomial sampling are discrete rather than continuous, you don't generally get exact 95% CI when you ask for 95% CI. The binom.coverage() function in the binom package shows what you actually get for CI coverage with each test. For my above example of 1% prevalence and a sample size of 387, the exact test favored by BruceET with a requested 95% CI gives 96.2% coverage (as do several other tests); the Wilson and logit tests give 93.7% coverage.

Third, as you seem to be designing a study, you should be thinking more directly in terms of power, as in BruceET's answer: what alternatives do you wish to distinguish from a value of 1% prevalence? As an extreme example, if prevalence is only 1%, how many samples do you need to examine to be assured of getting at least one positive sample? As the probability of failure (negative sample) is 0.99 for each sample, the probability of $n$ failures without any successes is $0.99^n$.

With 160 samples you thus have a 20% chance of no successes (alternatively, 80% power to find at least 1 positive sample), and even with 300 trials you still have a 4.9% chance of no successes. Or you could equivalently apply the rule of three: if there were no successes in 300 trials, your 95% CI would be [0,0.01]. So you clearly need at least 300 trials. How many more than that depends on how precisely you want to rule out the possibility of a specific larger probability (the alternative hypotheses in BruceET's answer).

Fourth, the point estimates and CI that you report will depend on how many positive samples you actually find. If you take 300 samples with a true success probability of 1% at each trial, then the probabilities of finding the following numbers of positive cases, the point estimates of the probability of a positive sample, and the corresponding nominal 95% CI for the "exact" binomial test would be:

# positive probability of finding that # estimated p(positive) lower 95% CI upper 95% CI
0 0.05 0 0 0.012
1 0.15 0.0033 ~0 0.018
2 0.22 0.0067 0.0008 0.024
3 0.23 0.0100 0.0021 0.029
4 0.17 0.0133 0.0036 0.034
5 0.10 0.0167 0.0054 0.038
6 0.05 0.0200 0.0074 0.043

For this particular scenario the "exact" test is conservative (coverage 98.9%), but that's mostly with respect to the lower limit. As you can see, the inherent randomness of binomial sampling will set the limit to the precision of what you report.

In summary, please don't just rely on a simple calculator for designing your study. Think carefully through the possibilities, how your assumptions might be in error, and how those aspects of the study would play out in practice.

$\endgroup$
0
0
$\begingroup$

You are right. This calculator uses a formula that is not appropriate for very small (or very large) effects.

If the incidence of the condition is 4%, then in a sample of 43 people, the likelihood of seeing 0,1, or 2 cases is 0.17, 0.31, or 0.27 respectively. So there is more than a 50% probability that, if your hypothesis was true, you would actually see one or more cases. If that was your outcome, you could not rule out an incidence greater than 1/43.

$\endgroup$
3
  • 1
    $\begingroup$ What formula can I use to estimate such a small effect? Im trying to determine how much sample I need to detect the 1% in the population. $\endgroup$
    – Daniel
    May 18, 2021 at 15:50
  • $\begingroup$ What do you mean by "detect"? This could mean e.g. "Disprove the hypothesis that incidence > 2%" or "Disprove the hypothesis that incidence < 0.5%". $\endgroup$ May 20, 2021 at 7:33
  • $\begingroup$ Once you have a null hypothesis, you can use the formula for the binomial distribution to make a table of the probability of $n$ observations for different sample sizes $N$. $\endgroup$ May 20, 2021 at 7:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.