Sample size for binomial distribution for rare events

Question

I would like to estimate the parameter p in a binomial process but need to determine the sample size required. I believe the event is fairly rare, and I have read the normal approximation may not be appropriate for rare events. What would be the best way to determine the required sample size to estimate p for rare events?

Answer 1

When you're down in the few percent range of rare events you might think in terms of a Poisson distribution, which has a very useful property: the variance of a Poisson rate equals its mean. If you have specified failure rates that you would like to distinguish, Gerald van Belle uses this property to propose a simple rule of thumb (page 40):

Suppose the means of samples from two Poisson populations are to be compared in a two-sample test. Let $\theta_0$ and $\theta_1$ be the means of the two populations. Then the required number of observations per sample is: $$n=\frac{4}{\left(\sqrt{\theta_0}-\sqrt{\theta_1}\right)^2}.$$

That provides 80% power to detect the difference at a significance level of p < 0.05. So say that you expected one transcriber to make 4 errors per 1000 words and another to make 9 errors per thousand words. You would need to evaluate $\frac{4}{(3-2)^2}*1000= 4000$ words from each transcriber to distinguish them, under those specifications.

Why power calculations use normal approximations

I checked this rule-of-thumb estimate for 4 versus 9 errors per 1000 words against the formal binomial sample-size estimates provided by the bsamsize function in the R Hmisc package, the allegedly "exact" test of inequality of proportions between two independent groups provided by the G*Power program, and Russ Lenth's Java applet (without the continuity correction). They estimated 4054, 4054 and 4055 per transcriber respectively, pretty close to the rule of thumb.

Then I looked at the details. All of these power-calculation tools use the normal approximation! G*Power, even when you select the "exact" option, forces use of the normal approximation for sample-size estimation and only does exact calculations for other options. If you force G*Power to do an exact calculation (with its otherwise useless "achieved power" option) for 4054 words and the above error rates, you find a very close result to the desired type-I-error-rate/power targets: p = 0.052, power = 0.813, by likelihood-ratio test.

Another way to express the Poisson rule of thumb shows why power calculations (unlike single-sample confidence intervals, for which the binom package in R provides 11 different methods) routinely use normal approximations.

Consider how many counts you can distinguish between two conditions with n=1, just one experiment for each condition. In the above example with 4000 words, we have power to distinguish 16 errors from 36 errors, not quite a two-fold difference. In general, that above formula provides power to distinguish $k^2$ errors from $(k+2)^2$ errors, $k=4$ in this case. The next example with integer $k=5$ would finally provide power to find a two-fold difference, 25 versus 49 expected errors.

Then, recall that the normal approximation works pretty well if the total number of counts is greater than 5 or 10. This answer illustrates, and provides Python code for further exploration. But if you want to be able to detect differences in proportions on the order of two-fold or less, we just saw that you need to have 25 or more counts in the less frequent condition. When you have enough power to detect a two-fold difference, you are already in the range where the normal approximation to the binomial is OK.

In this particular practical situation, uncertainties from differences among speakers and among those doing the transcriptions among the vendors being evaluated should greatly outweigh any very minor distinction between normal-approximation and exact power calculations.