What is the best technique to calculate a confidence interval of a binomial experiment, if your estimate is that $p=0$ (or similarly $p=1$) and sample size is relatively small, for example $n=25$?

  • $\begingroup$ How close to zero is $\hat{p}$? Is it zero often, or on the order of 0.001, or 0.01, or ...? And how much data do you have? $\endgroup$ – jbowman Oct 9 '18 at 17:06
  • $\begingroup$ We usually have greater than 800 trials. We usually expect 0 to 0.1 for $\hat{p}$ $\endgroup$ – AI2.0 Oct 9 '18 at 17:21
  • $\begingroup$ Use Clopper–Pearson interval you linked. The general principle: Try Clopper–Pearson interval first. If computer cannot get the answer, try the approximation method, such as normal approximation. According to the current computer speed, I do not think we need approximation on most situations. $\endgroup$ – user158565 Oct 9 '18 at 17:27
  • $\begingroup$ For only getting the upper limit of the confidence interval with (1-$\alpha$ confidence level, we will just use B(1−$\alpha$;x+1,n−x) where x is the number of successes (or failures), n is the sample size. In python, we just use scipy.stats.beta.ppf(1−$\alpha$;x+1,n−x) . If this is TRUE, can we conclude that we are 1−$\alpha$ confident that the upper limit is bounded by the value we calculate from scipy.stats.beta.ppf(1−$\alpha$;x+1,n−x) ? $\endgroup$ – AI2.0 Oct 9 '18 at 18:19
  • 1
    $\begingroup$ With 800 trials, the usual Normal approximation will work reasonably well down to about $p=0.015$ (my simulations indicated a 94.5% actual coverage of a 95% confidence interval.) At 1000 trials and $p=0.01$, the actual coverage was about 92.7% (all based on 100,000 replications.) So this is only an issue for very low $p$, given your trial count. $\endgroup$ – jbowman Oct 9 '18 at 18:23

Do not use the normal approximation

Much has been written about this problem. A general advice is to never use the normal approximation (i.e., the asymptotic/Wald confidence interval), as it has terrible coverage properties. R code for illustrating this:

p = seq(0,1,.001)
coverage = binom.coverage(p, 25, method="asymptotic")$coverage
plot(p, coverage, type="l")
abline(h=.95, col="red")

Coverage probabilities for asymptotic confidence intervals for a binomial proportion.

For small success probabilities, you might ask for a 95% confidence interval, but actually get, say, a 10% confidence interval!


So what should we use? I believe the current recommendations are the ones listed in the paper Interval Estimation for a Binomial Proportion by Brown, Cai and DasGupta in Statistical Science 2001, vol. 16, no. 2, pages 101–133. The authors examined several methods for calculating confidence intervals, and came to the following conclusion.

[W]e recommend the Wilson interval or the equal-tailed Jeffreys prior interval for small n and the interval suggested in Agresti and Coull for larger n.

The Wilson interval is also sometimes called the score interval, since it’s based on inverting a score test.

Calculating the intervals

To calculate these confidence intervals, you can use this online calculator or the binom.confint() function in the binom package in R. For example, for 0 successes in 25 trials, the R code would be:

> binom.confint(0, 25, method=c("wilson", "bayes", "agresti-coull"),
         method x  n  mean  lower upper
1 agresti-coull 0 25 0.000 -0.024 0.158
2         bayes 0 25 0.019  0.000 0.073
3        wilson 0 25 0.000  0.000 0.133

Here bayes is the Jeffreys interval. (The argument type="central" is needed to get the equal-tailed interval.)

Note that you should decide on which of the three methods you want to use before calculating the interval. Looking at all three and selecting the shortest will naturally give you too small coverage probability.

A quick, approximate answer

As a final note, if you observe exactly zero successes in your n trials and just want a very quick approximate confidence interval, you can use the rule of three. Simply divide the number 3 by n. In the above example n is 25, so the upper bound is 3/25 = 0.12 (the lower bound is of course 0).

| cite | improve this answer | |
  • 2
    $\begingroup$ I wouldn’t say that you can conclude it with ‘95% certainty’ (Google for ‘correct interpretation of confidence intervals’). Also, this is based on the assumption of independent trials with equal success probabilities, which may not be realistic here. Perhaps the last panels installed had a higher risk of being incorrectly installed (the person installing them was getting tired/bored). Or perhaps the first ones were, since the person was less experienced then. Anyway, if the architect was told to test if all the panels are correctly installed, he should do his job, not just test a sample! $\endgroup$ – Karl Ove Hufthammer Jan 19 '14 at 11:52
  • 7
    $\begingroup$ bayes uses the uniform prior (instead of Jeffrey's) when both shape parameters are 1. I emailed with the maintainer of the binom package out of curiosity about the (dis)advantages of Jeffrey's vs. uniform prior and he told me that a new version will use the uniform prior as default. So don't wonder if the results vary slightly in future. $\endgroup$ – cbeleites unhappy with SX Mar 2 '14 at 20:39
  • 4
    $\begingroup$ This is an excellent answer. It conveys all the key information you can read in papers on the topic, but very concisely and clearly. If I could upvote twice I would. $\endgroup$ – SigmaX Jun 18 '15 at 7:07
  • 7
    $\begingroup$ The binconf method in Hmisc also computes these intervals. It defaults to the Wilson method. $\endgroup$ – SigmaX Jun 18 '15 at 7:09
  • 1
    $\begingroup$ A more recent paper with a review of about a dozen methods (and references to another thirty, maybe) is doi.org/10.1093/jssam/smv024. (Should this be folded into the answer as an update @KarlOveHufthammer?) $\endgroup$ – StasK Sep 19 '17 at 14:14

Agretsi (2007, pp.9-10) shows that when a proportion falls near 0 or 1, the confidence interval $p\pm z_{\alpha/2}\sqrt{p(1-p)/n}$ performs poorly. Instead, use a "duality wuith significance tests... [that] consists of all values of $\pi_0$ for the null hypothesis parameter that a judged plausible," where $\pi_0$ is the unknown parameter. Do this by solving for $\pi_0$ in the equation $$\frac{|p-\pi_0|}{\sqrt{p(1-p)/n}}=0$$. Do this by squaring both sides, yielding $$(1+z_0^2/n)\pi_0^2+(-2p-z_0^2/n)\pi_0+p^2=0$$ Solve using the quadratic formula, which will yield the appropriate critical z-value.

| cite | improve this answer | |
  • 1
    $\begingroup$ Thank you for the notes. Just want to clarify: $\pi_0$ is the assumed failure (or success) rate in the population whereas p is the observed failure (or success rate) from the sample. And n is the sample size, so we are trying to solve the approximate z-value? (what are the underlying assumptions here?) (Would you mind link me to the paper Agretsi (2007, pp.9-10) ). $\endgroup$ – AI2.0 Oct 9 '18 at 17:25
  • $\begingroup$ Yes, $\pi_0$ is the population parameter, $p$ is the parameter estimate based on your sample, and $n$ is the sample size. This procedure will give you the critical z-value you want. The underlying assumptions are fleshed out in Agretsi and Coull (1998), link at the end. Unfortunately, Agretsi (2007) is a textbook, so I cannot link to it. scholar.google.com/… $\endgroup$ – Jay Schyler Raadt Oct 9 '18 at 19:03
  • 2
    $\begingroup$ That’s Agresti. $\endgroup$ – Nick Cox Oct 9 '18 at 19:51
  • $\begingroup$ @NickCox it's a different work $\endgroup$ – Jay Schyler Raadt Oct 10 '18 at 1:18
  • 1
    $\begingroup$ Alan Agresti has published various texts. I guess you are alluding to An Introduction to Categorical Data Analysis (2nd edition 2007; 3rd edition scheduled for October 2018 publication and may carry a 2019 date) from John Wiley. $\endgroup$ – Nick Cox Oct 10 '18 at 7:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.