I know the standard formula for the Bernoulli CI is:

$$\hat{p}\pm z_{1-\alpha/2}\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$$

If $\hat{p} = \frac{m}{n}$ how do I estimate the confidence interval when$\ n$ is small and$\ m = 0$? This case would collapse the above equation to $\ 0 \pm 0$, which implies that the confidence interval does not improve with larger $\ n$.

In my mind, the CI should start at [0,1] and the upper bound should decrease as $\ n$ increases, given that $\ m$ remains at 0.

  • 1
    $\begingroup$ You could use the actual likelihood function of your data L(p) which will be proportional to $p^m\,(1-p)^{n-m}$ Given some prior on p, for example some Beta distribution you can get the posterior and the credible intervals on p. $\endgroup$ – sega_sai Nov 2 '16 at 18:09
  • $\begingroup$ Notice that the same applies to $\hat{p}=1$. $\endgroup$ – Alexis Nov 2 '16 at 18:27
  • $\begingroup$ Running a Bayesian analysis of the data will produce a credible interval even when $\hat{p}=0$. $\endgroup$ – Xi'an Nov 2 '16 at 20:00
  • $\begingroup$ One possibility that's sometimes used in some application areas is called "the rule of three". Also see Wikipedia's page on it $\endgroup$ – Glen_b -Reinstate Monica Nov 2 '16 at 21:28

The reason the usual "CLT" confidence interval becomes 0 is because when $p$ is very close to 0 or 1 (and the relative number of samples is low), the CLT becomes a bad approximation. This is because when $p=0,1$, your random variable is constant. On the other hand, when $p$ is very close to 1 or 0, you need a very large amount of samples to distinguish $p$ from exactly 1 or 0.

There are a couple of approaches to get the true confidence interval. The easy way is to appeal to the Wilson score interval:

$$\frac{1}{1 + \frac{1}{n} z^2} \left[ \hat{p} + \frac{1}{2n} z^2 \pm z \sqrt{ \frac{1}{n}\hat{p} \left(1 - \hat{p}\right) + \frac{1}{4n^2}z^2 } \right].$$

The second option is to numerically estimate the true confidence interval by explicitly using the binomial distribution, as opposed to appealing to the normal distribution.

  • $\begingroup$ Thanks for the help! The Wilson score interval behaves closer to my expectations. I'm going to do some reading on these different methods for confidence intervals, as all I've been exposed to are the CLT methods. $\endgroup$ – japata Nov 2 '16 at 19:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.