1
$\begingroup$

Suppose you have n Bernoulli trials and zero observed successes. The lower bound on a $1-\alpha$ two-sided confidence interval for the proportion of successes will of course be zero. By the definition of the Clopper-Pearson interval, the upper bound is $\{ \theta | P[Bin(n, \theta) \geq 0] \geq \alpha/2 \}$.

My question is: isn't the coverage of this interval $1-\alpha/2$ instead of $1-\alpha$?

Note: In R, binom::binom.confint returns an upper bound such that $ (1 - UB)^n = \alpha/2 $ when a $1 - \alpha$ confidence level is specified.

Improve this question
$\endgroup$
4
  • $\begingroup$ How are you calculating the variance if you have no successes? $\endgroup$
    – Dave
    Commented May 5, 2020 at 16:43
  • 1
    $\begingroup$ @Dave, Clopper-Pearson intervals don't require variance calculations - they work fine for zero successes or zero failures. $\endgroup$
    – knrumsey
    Commented May 5, 2020 at 17:14
  • $\begingroup$ A confidence interval is a procedure. Its coverage does not depend on the data. $\endgroup$
    – whuber ♦
    Commented May 5, 2020 at 17:26
  • $\begingroup$ An Agresti-Coull 'plus-4' interval would use the estimate $\tilde p = (0+2)/(n+4)$ for the probability of Success, and its standard error is estimated as $\tilde{SE}=\sqrt{\tilde p(1-\tilde p)/(n+4)}.$ Then using norm aprx to bino, 2-sided a 95% CI is $\tilde p \pm 1.96\tilde{SE}.$ Often the A-C interval stays inside $(0,1),$ when the Wald CI does not. // If you want a 95% one-sided CI, then put the full 'error 5%' in the one tail. $\endgroup$
    – BruceET
    Commented May 5, 2020 at 20:30

1 Answer 1

Reset to default
1
$\begingroup$

Any frequentist confidence interval based on a normal approximation may give puzzling results when there are $0$ successes in $n$ trials.

It might be better to get a Bayesian probability interval estimate (sometimes called 'credible interval') based on a suitable beta prior distribution. [The Bayesian approach to inference treats the success probability $p$ as a random variable--with a prior distribution to start, and finally a posterior distribution to reflect the prior distribution and the data.]

If you have little or no prior information, you might use $\mathsf{Unif}(0,1) \equiv \mathsf{Beta}(1,1)$ as the prior. An alternative would be to use the Jeffreys prior $\mathsf{Beta}(.5, .5).$

If getting no successes does not come as a surprise, then perhaps you want to use a prior distribution on the success probability that puts more probability toward the lower end of $(0,1).$

Suppose you use a uniform prior and get $x = 0$ successes in $n = 20$ trials. Then the posterior distribution is $\mathsf{Beta}(1+x, 1+n-x) = \mathsf{Beta}(1,21).$ Then a one-sided 95% Bayesian interval estimate $(0, 0.0325)$ can be found in R as shown below. This interval suggests that the small success probability is likely below $0.0325.$

qbeta(.5, 1, 21)
[1] 0.03246822

If you really want a two-sided interval estimate for the binomial success probability, then you could use $(0.0012, 0.1611).$ [The Jeffries 95% two-sided interval is $(0.000024, 0.11664).]$

qbeta(c(.025, .975), 1, 21)    # uniform prior
[1] 0.001204883 0.161097615

qbeta(c(.025, .975), .5, 20.5) # Jeffries prior
[1] 2.424648e-05 1.166390e-01

This interval makes it clear that you are not suggesting that the events in question are completely impossible. For example, if the task is to estimate the proportion of cattle in Montana that have a particular disease, you are not saying that negative results from 20 randomly chosen Montana animals should be taken as an indication that cattle throughout Montana are totally free of the disease.

There are important philosophical differences between Bayesian and frequentist statisticians, and these differences extend to how interval estimates are interpreted by both groups of statisticians. However, frequentist statisticians are often happy to use Bayesian computational methods for getting an interval estimate, and then calling the result a 'confidence interval' to be given a frequentist interpretation.

Note: For an extensive list of types of frequentist CIs for the binomial success probability see Wikipedia, where assumptions are discussed. Notice that the Jeffreys interval, interpreted as frequentist, is among the first types of CIs discussed there.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for this. I agree that the Bayesian approach seems particularly suited to this sort of situation. But I'm most interested in the Clopper-Pearson or Wilson confidence intervals in particular. $\endgroup$
    – Carter
    Commented May 5, 2020 at 20:09
  • $\begingroup$ Wikipedia: "The Clopper–Pearson interval is an exact interval since it is based directly on the binomial distribution rather than any approximation to the binomial distribution. This interval never has less than the nominal coverage for any population proportion, but that means that it is usually conservative." This means that the interval is often unnecessarily long. // For 95% CIs, the Wilson interval is well-approximated by the Agresti interval, but both rely on norm aprx to bino, so neither works well for small $n$ or estimates very close to $0.$ // Be sure your 'interest' is well-chosen. $\endgroup$
    – BruceET
    Commented May 5, 2020 at 20:37
  • $\begingroup$ In my case, where n=30, the Agresti-Coull interval is wider than Wilson, Clopper-Pearson, and Jeffrey's methods (with Jeffrey's being the shortest). In fact, for 0 successes and n trials, it appears that Agresti-Coull is only shorter than Clopper-Pearson for very small n. Am I missing something here? (I know, in practice, it's not advisable to calculate CIs using many methods and pick the shortest. This is just for learning.) $\endgroup$
    – Carter
    Commented May 5, 2020 at 22:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.