I would like to estimate the parameter p in a binomial process but need to determine the sample size required. I believe the event is fairly rare, and I have read the normal approximation may not be appropriate for rare events. What would be the best way to determine the required sample size to estimate p for rare events?

  • $\begingroup$ Just how low do you expect $p$ to be, and what precision do you need in the estimate? $\endgroup$ – EdM Oct 5 at 14:58
  • $\begingroup$ I'll be sampling from a few different groups. I'd estimate p at about 2-10%, depending on the demographic i'm sampling from. I'm looking for a precision of 1%. $\endgroup$ – user6883405 Oct 5 at 15:25
  • $\begingroup$ Do you want to be able to distinguish the values of $p$ among the groups? If so, how many groups? Is all that you care about the value of $p$ within each group, or are you also trying to correct for covariates (age, sex, socioeconomic status,...), perhaps trying to model $p$ as a function of several variables? $\endgroup$ – EdM Oct 5 at 16:01

When you're down in the few percent range of rare events you might think in terms of a Poisson distribution, which has a very useful property: the variance of a Poisson rate equals its mean. If you have specified failure rates that you would like to distinguish, Gerald van Belle uses this property to propose a simple rule of thumb (page 40):

Suppose the means of samples from two Poisson populations are to be compared in a two-sample test. Let $\theta_0$ and $\theta_1$ be the means of the two populations. Then the required number of observations per sample is: $$n=\frac{4}{\left(\sqrt{\theta_0}-\sqrt{\theta_1}\right)^2}.$$

That provides 80% power to detect the difference at a significance level of p < 0.05. So say that you expected one transcriber to make 4 errors per 1000 words and another to make 9 errors per thousand words. You would need to evaluate $\frac{4}{(3-2)^2}*1000= 4000$ words from each transcriber to distinguish them, under those specifications.

Why power calculations use normal approximations

I checked this rule-of-thumb estimate for 4 versus 9 errors per 1000 words against the formal binomial sample-size estimates provided by the bsamsize function in the R Hmisc package, the allegedly "exact" test of inequality of proportions between two independent groups provided by the G*Power program, and Russ Lenth's Java applet (without the continuity correction). They estimated 4054, 4054 and 4055 per transcriber respectively, pretty close to the rule of thumb.

Then I looked at the details. All of these power-calculation tools use the normal approximation! G*Power, even when you select the "exact" option, forces use of the normal approximation for sample-size estimation and only does exact calculations for other options. If you force G*Power to do an exact calculation (with its otherwise useless "achieved power" option) for 4054 words and the above error rates, you find a very close result to the desired type-I-error-rate/power targets: p = 0.052, power = 0.813, by likelihood-ratio test.

Another way to express the Poisson rule of thumb shows why power calculations (unlike single-sample confidence intervals, for which the binom package in R provides 11 different methods) routinely use normal approximations.

Consider how many counts you can distinguish between two conditions with n=1, just one experiment for each condition. In the above example with 4000 words, we have power to distinguish 16 errors from 36 errors, not quite a two-fold difference. In general, that above formula provides power to distinguish $k^2$ errors from $(k+2)^2$ errors, $k=4$ in this case. The next example with integer $k=5$ would finally provide power to find a two-fold difference, 25 versus 49 expected errors.

Then, recall that the normal approximation works pretty well if the total number of counts is greater than 5 or 10. This answer illustrates, and provides Python code for further exploration. But if you want to be able to detect differences in proportions on the order of two-fold or less, we just saw that you need to have 25 or more counts in the less frequent condition. When you have enough power to detect a two-fold difference, you are already in the range where the normal approximation to the binomial is OK.

In this particular practical situation, uncertainties from differences among speakers and among those doing the transcriptions among the vendors being evaluated should greatly outweigh any very minor distinction between normal-approximation and exact power calculations.

| cite | improve this answer | |
  • $\begingroup$ Thank you! How might I modify that rule of thumb for a higher level of power than 80%? I found a binomial sample size estimator implemented in R (binomSamSize) and used the Fosgate estimation approach. The actual Fosgate paper (Fosgate 2005) mentions that it is a less conservative approach than the Clopper–Pearson estimation approach. Are you aware of a Clopper–Pearson approach to estimating sample size in R? $\endgroup$ – user6883405 Oct 5 at 19:48
  • $\begingroup$ @user6883405 if you look at the (freely available) chapter by van Belle linked in the answer, you will see that the numerator of 4 in that rule of thumb comes from dividing a more generic numerator of 16 (for 2-sample comparisons) in Table 2.1 by a factor of 4. For 90% power, use instead a numerator of 21/4 or 5.25; for 95% power, use 26/4 = 6.5; for 97.5% power, 31/4 = 7.75, based on values in Table 2.1. More on other approaches when I get a chance later. $\endgroup$ – EdM Oct 5 at 21:46
  • $\begingroup$ @user6883405 elaborated more on why the normal approximation is OK for most practical power calculations. $\endgroup$ – EdM Oct 6 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.