Confidence interval around binomial estimate of 0 or 1

Question

What is the best technique to calculate a confidence interval of a binomial experiment, if your estimate is that $p=0$ (or similarly $p=1$) and sample size is relatively small, for example $n=25$?

Answer 1

Do not use the normal approximation

Much has been written about this problem. A general advice is to never use the normal approximation (i.e., the asymptotic/Wald confidence interval), as it has terrible coverage properties. R code for illustrating this:

library(binom)
p = seq(0,1,.001)
coverage = binom.coverage(p, 25, method="asymptotic")$coverage
plot(p, coverage, type="l")
binom.confint(0,25)
abline(h=.95, col="red")

For small success probabilities, you might ask for a 95% confidence interval, but actually get, say, a 10% confidence interval!

Recommendations

So what should we use? I believe the current recommendations are the ones listed in the paper Interval Estimation for a Binomial Proportion by Brown, Cai and DasGupta in Statistical Science 2001, vol. 16, no. 2, pages 101–133. The authors examined several methods for calculating confidence intervals, and came to the following conclusion.

[W]e recommend the Wilson interval or the equal-tailed Jeffreys prior interval for small n and the interval suggested in Agresti and Coull for larger n.

The Wilson interval is also sometimes called the score interval, since it’s based on inverting a score test.

Calculating the intervals

To calculate these confidence intervals, you can use this online calculator or the binom.confint() function in the binom package in R. For example, for 0 successes in 25 trials, the R code would be:

> binom.confint(0, 25, method=c("wilson", "bayes", "agresti-coull"),
  type="central")
         method x  n  mean  lower upper
1 agresti-coull 0 25 0.000 -0.024 0.158
2         bayes 0 25 0.019  0.000 0.073
3        wilson 0 25 0.000  0.000 0.133

Here bayes is the Jeffreys interval. (The argument type="central" is needed to get the equal-tailed interval.)

Note that you should decide on which of the three methods you want to use before calculating the interval. Looking at all three and selecting the shortest will naturally give you too small coverage probability.

A quick, approximate answer

As a final note, if you observe exactly zero successes in your n trials and just want a very quick approximate confidence interval, you can use the rule of three. Simply divide the number 3 by n. In the above example n is 25, so the upper bound is 3/25 = 0.12 (the lower bound is of course 0).

Answer 2

The Wilson score interval gives a good answer in this (and other) cases

Confidence intervals for a binomial probability parameter can be formed using the Wilson score interval and this interval respects the allowable parameter range for the probability parameter. It also accommodates the extreme case where all observations are in one of the binary categories. This case is discussed in O'Neill (2021) (p. 5), which also outlines some other properties of these intervals. When we have $n$ data points with sample mean $\bar{x}=0$ or $\bar{x}=1$ the respective confidence intervals with confidence level $1-\alpha$ are given by:

$$\begin{align} \text{CI}(1-\alpha, \bar{x}=0) &= \bigg[ 0, \frac{\chi_\alpha^2}{n+\chi_\alpha^2} \bigg], \\[12pt] \text{CI}(1-\alpha, \bar{x}=1) &= \bigg[ \frac{n}{n+\chi_\alpha^2}, 1 \bigg], \\[12pt] \end{align}$$

where $\chi_\alpha^2$ is the critical point corresponding to an upper tail area of $\alpha$ for the chi-squared distribution with one degree-of-freedom. Using a 95% confidence level gives $\chi_\alpha^2 = 3.841459$ which then leads to the following intervals that are slight variations of the "rule of three":

$$\begin{align} \text{CI}(0.95, \bar{x}=0) &= \bigg[ 0, \frac{3.841459}{n+3.841459} \bigg], \\[12pt] \text{CI}(0.95, \bar{x}=1) &= \bigg[ \frac{n}{n+3.841459}, 1 \bigg]. \\[12pt] \end{align}$$

Answer 3

Agresti (2007, pp.9-10) shows that when a proportion falls near 0 or 1, the confidence interval $p\pm z_{\alpha/2}\sqrt{p(1-p)/n}$ performs poorly. Instead, use a "duality with significance tests... [that] consists of all values of $\pi_0$ for the null hypothesis parameter that a judged plausible," where $\pi_0$ is the unknown parameter. Do this by solving for $\pi_0$ in the equation $$\frac{|p-\pi_0|}{\sqrt{p(1-p)/n}}=0$$. Do this by squaring both sides, yielding $$(1+z_0^2/n)\pi_0^2+(-2p-z_0^2/n)\pi_0+p^2=0$$ Solve using the quadratic formula, which will yield the appropriate critical z-value.