# Confidence interval around binomial estimate of 0 or 1

What is the best technique to calculate a confidence interval of a binomial experiment, if your estimate is that $p=0$ (or similarly $p=1$) and sample size is relatively small, for example $n=25$?

• How close to zero is $\hat{p}$? Is it zero often, or on the order of 0.001, or 0.01, or ...? And how much data do you have? Oct 9, 2018 at 17:06
• We usually have greater than 800 trials. We usually expect 0 to 0.1 for $\hat{p}$ Oct 9, 2018 at 17:21
• Use Clopper–Pearson interval you linked. The general principle: Try Clopper–Pearson interval first. If computer cannot get the answer, try the approximation method, such as normal approximation. According to the current computer speed, I do not think we need approximation on most situations. Oct 9, 2018 at 17:27
• For only getting the upper limit of the confidence interval with (1-$\alpha$ confidence level, we will just use B(1−$\alpha$;x+1,n−x) where x is the number of successes (or failures), n is the sample size. In python, we just use scipy.stats.beta.ppf(1−$\alpha$;x+1,n−x) . If this is TRUE, can we conclude that we are 1−$\alpha$ confident that the upper limit is bounded by the value we calculate from scipy.stats.beta.ppf(1−$\alpha$;x+1,n−x) ? Oct 9, 2018 at 18:19
• With 800 trials, the usual Normal approximation will work reasonably well down to about $p=0.015$ (my simulations indicated a 94.5% actual coverage of a 95% confidence interval.) At 1000 trials and $p=0.01$, the actual coverage was about 92.7% (all based on 100,000 replications.) So this is only an issue for very low $p$, given your trial count. Oct 9, 2018 at 18:23

# Do not use the normal approximation

Much has been written about this problem. A general advice is to never use the normal approximation (i.e., the asymptotic/Wald confidence interval), as it has terrible coverage properties. R code for illustrating this:

library(binom)
p = seq(0,1,.001)
coverage = binom.coverage(p, 25, method="asymptotic")$coverage plot(p, coverage, type="l") binom.confint(0,25) abline(h=.95, col="red")  For small success probabilities, you might ask for a 95% confidence interval, but actually get, say, a 10% confidence interval! # Recommendations So what should we use? I believe the current recommendations are the ones listed in the paper Interval Estimation for a Binomial Proportion by Brown, Cai and DasGupta in Statistical Science 2001, vol. 16, no. 2, pages 101–133. The authors examined several methods for calculating confidence intervals, and came to the following conclusion. [W]e recommend the Wilson interval or the equal-tailed Jeffreys prior interval for small n and the interval suggested in Agresti and Coull for larger n. The Wilson interval is also sometimes called the score interval, since it’s based on inverting a score test. # Calculating the intervals To calculate these confidence intervals, you can use this online calculator or the binom.confint() function in the binom package in R. For example, for 0 successes in 25 trials, the R code would be: > binom.confint(0, 25, method=c("wilson", "bayes", "agresti-coull"), type="central") method x n mean lower upper 1 agresti-coull 0 25 0.000 -0.024 0.158 2 bayes 0 25 0.019 0.000 0.073 3 wilson 0 25 0.000 0.000 0.133  Here bayes is the Jeffreys interval. (The argument type="central" is needed to get the equal-tailed interval.) Note that you should decide on which of the three methods you want to use before calculating the interval. Looking at all three and selecting the shortest will naturally give you too small coverage probability. # A quick, approximate answer As a final note, if you observe exactly zero successes in your n trials and just want a very quick approximate confidence interval, you can use the rule of three. Simply divide the number 3 by n. In the above example n is 25, so the upper bound is 3/25 = 0.12 (the lower bound is of course 0). • I wouldn’t say that you can conclude it with ‘95% certainty’ (Google for ‘correct interpretation of confidence intervals’). Also, this is based on the assumption of independent trials with equal success probabilities, which may not be realistic here. Perhaps the last panels installed had a higher risk of being incorrectly installed (the person installing them was getting tired/bored). Or perhaps the first ones were, since the person was less experienced then. Anyway, if the architect was told to test if all the panels are correctly installed, he should do his job, not just test a sample! Jan 19, 2014 at 11:52 • bayes uses the uniform prior (instead of Jeffrey's) when both shape parameters are 1. I emailed with the maintainer of the binom package out of curiosity about the (dis)advantages of Jeffrey's vs. uniform prior and he told me that a new version will use the uniform prior as default. So don't wonder if the results vary slightly in future. Mar 2, 2014 at 20:39 • This is an excellent answer. It conveys all the key information you can read in papers on the topic, but very concisely and clearly. If I could upvote twice I would. Jun 18, 2015 at 7:07 • The binconf method in Hmisc also computes these intervals. It defaults to the Wilson method. Jun 18, 2015 at 7:09 • A more recent paper with a review of about a dozen methods (and references to another thirty, maybe) is doi.org/10.1093/jssam/smv024. (Should this be folded into the answer as an update @KarlOveHufthammer?) Sep 19, 2017 at 14:14 Agresti (2007, pp.9-10) shows that when a proportion falls near 0 or 1, the confidence interval $$p\pm z_{\alpha/2}\sqrt{p(1-p)/n}$$ performs poorly. Instead, use a "duality with significance tests... [that] consists of all values of $$\pi_0$$ for the null hypothesis parameter that a judged plausible," where $$\pi_0$$ is the unknown parameter. Do this by solving for $$\pi_0$$ in the equation $$\frac{|p-\pi_0|}{\sqrt{p(1-p)/n}}=0$$. Do this by squaring both sides, yielding $$(1+z_0^2/n)\pi_0^2+(-2p-z_0^2/n)\pi_0+p^2=0$$ Solve using the quadratic formula, which will yield the appropriate critical z-value. • Thank you for the notes. Just want to clarify:$\pi_0$is the assumed failure (or success) rate in the population whereas p is the observed failure (or success rate) from the sample. And n is the sample size, so we are trying to solve the approximate z-value? (what are the underlying assumptions here?) (Would you mind link me to the paper Agretsi (2007, pp.9-10) ). Oct 9, 2018 at 17:25 • Yes,$\pi_0$is the population parameter,$p$is the parameter estimate based on your sample, and$n\$ is the sample size. This procedure will give you the critical z-value you want. The underlying assumptions are fleshed out in Agretsi and Coull (1998), link at the end. Unfortunately, Agretsi (2007) is a textbook, so I cannot link to it. scholar.google.com/… Oct 9, 2018 at 19:03
• That’s Agresti. Oct 9, 2018 at 19:51
• Alan Agresti has published various texts. I guess you are alluding to An Introduction to Categorical Data Analysis (2nd edition 2007; 3rd edition scheduled for October 2018 publication and may carry a 2019 date) from John Wiley. Oct 10, 2018 at 7:31
• This is called Wilson confidence interval. Jun 16, 2020 at 17:05

#### The Wilson score interval gives a good answer in this (and other) cases

Confidence intervals for a binomial probability parameter can be formed using the Wilson score interval and this interval respects the allowable parameter range for the probability parameter. It also accommodates the extreme case where all observations are in one of the binary categories. This case is discussed in O'Neill (2021) (p. 5), which also outlines some other properties of these intervals. When we have $$n$$ data points with sample mean $$\bar{x}=0$$ or $$\bar{x}=1$$ the respective confidence intervals with confidence level $$1-\alpha$$ are given by:

\begin{align} \text{CI}(1-\alpha, \bar{x}=0) &= \bigg[ 0, \frac{\chi_\alpha^2}{n+\chi_\alpha^2} \bigg], \\[12pt] \text{CI}(1-\alpha, \bar{x}=1) &= \bigg[ \frac{n}{n+\chi_\alpha^2}, 1 \bigg], \\[12pt] \end{align}

where $$\chi_\alpha^2$$ is the critical point corresponding to an upper tail area of $$\alpha$$ for the chi-squared distribution with one degree-of-freedom. Using a 95% confidence level gives $$\chi_\alpha^2 = 3.841459$$ which then leads to the following intervals that are slight variations of the "rule of three":

\begin{align} \text{CI}(0.95, \bar{x}=0) &= \bigg[ 0, \frac{3.841459}{n+3.841459} \bigg], \\[12pt] \text{CI}(0.95, \bar{x}=1) &= \bigg[ \frac{n}{n+3.841459}, 1 \bigg]. \\[12pt] \end{align}