Sample size calculation for estimating a small population proportion I'm having a disconnect on power and sample size estimation. I'm using the sample size calculator here: https://select-statistics.co.uk/calculators/sample-size-calculator-population-proportion/
Im trying to estimate how much sample I need to estimate a population with a small proportion. Say the population proportion for cancer is 1% and I want to be 95% confident that my error is 3%. I would imagine that I would need a large sample size if the population proportion is 1%, but the calculator gives me n=43. I don't understand how I could reasonably estimate such a small proportion with n=43. If only 1 out of every 100 folks in the population have cancer, how could I detect it with a sample of 43 people?
 A: A few thoughts beyond what BruceET provided in an extensive answer (+1).
First, the calculator page you link itself warns with respect to their method:

If, the sample proportion is close to 0 or 1 then this approximation is not valid and you need to consider an alternative sample size calculation method.

A reason is that the normal-approximation formula used on that page assumes symmetry in potential results around the estimated prevalence. If you are allowing for +/- 3 percentage points error around an estimated 1% population prevalence, the lower limit of 0 in probability means that you've destroyed that symmetry; you can't get a -2% probability.
If you use that same calculator and instead ask for +/- 0.99 percentage point error around the 1% estimated prevalence (that is, allowing a range between 0.01% and 1.99%), you get an estimated sample size of 387. That's probably more in line with what you were expecting to find.
Second, there is a large number of ways to estimate binomial proportion confidence intervals. The R binom package provides 11 of them. As the results of binomial sampling are discrete rather than continuous, you don't generally get exact 95% CI when you ask for 95% CI. The binom.coverage() function in the binom package shows what you actually get for CI coverage with each test. For my above example of 1% prevalence and a sample size of 387, the exact test favored by BruceET with a requested 95% CI gives 96.2% coverage (as do several other tests); the Wilson and logit tests give 93.7% coverage.
Third, as you seem to be designing a study, you should be thinking more directly in terms of power, as in BruceET's answer: what alternatives do you wish to distinguish from a value of 1% prevalence? As an extreme example, if prevalence is only 1%, how many samples do you need to examine to be assured of getting at least one positive sample? As the probability of failure (negative sample) is 0.99 for each sample, the probability of $n$ failures without any successes is $0.99^n$.
With 160 samples you thus have a 20% chance of no successes (alternatively, 80% power to find at least 1 positive sample), and even with 300 trials you still have a 4.9% chance of no successes. Or you could equivalently apply the rule of three: if there were no successes in 300 trials, your 95% CI would be [0,0.01]. So you clearly need at least 300 trials. How many more than that depends on how precisely you want to rule out the possibility of a specific larger probability (the alternative hypotheses in BruceET's answer).
Fourth, the point estimates and CI that you report will depend on how many positive samples you actually find. If you take 300 samples with a true success probability of 1% at each trial, then the probabilities of finding the following numbers of positive cases, the point estimates of the probability of a positive sample, and the corresponding nominal 95% CI for the "exact" binomial test would be:




# positive
probability of finding that #
estimated p(positive)
lower 95% CI
upper 95% CI




0
0.05
0
0
0.012


1
0.15
0.0033
~0
0.018


2
0.22
0.0067
0.0008
0.024


3
0.23
0.0100
0.0021
0.029


4
0.17
0.0133
0.0036
0.034


5
0.10
0.0167
0.0054
0.038


6
0.05
0.0200
0.0074
0.043




For this particular scenario the "exact" test is conservative (coverage 98.9%), but that's mostly with respect to the lower limit. As you can see, the inherent randomness of binomial sampling will set the limit to the precision of what you report.
In summary, please don't just rely on a simple calculator for designing your study. Think carefully through the possibilities, how your assumptions might be in error, and how those aspects of the study would play out in practice.
A: You are right. This calculator uses a formula that is not appropriate for very small (or very large) effects.
If the incidence of the condition is 4%, then in a sample of 43 people, the likelihood of seeing 0,1, or 2 cases is  0.17, 0.31, or 0.27 respectively. So there is more than a 50% probability that, if your hypothesis was true, you would actually see one or more cases. If that was your outcome, you could not rule out an incidence greater than 1/43.
