How many counts for good frequency estimate (binomial)?

Question

This is a pretty basic question, I think, but I'm finding it difficult to locate an answer or decent treatment of it on the web.

For a binary dataset (binomial) with some unknown parameter p (probability of "heads" or success or something observed), how does the accuracy of the estimated p depend on the number of trials?

I've read a number of ways to calculate confidence intervals around the estimated p, but these seem to all depend on assuming a large number of trials and a normal distribution. I'm interested in particular here with small numbers of trials.

Thanks for any solutions or pointers.

Answer 1

This actually has a large literature. A good entry point is Agresti & Coull. They recommend one simple estimate which is to add 2 failures and two successes (or heads and tails or whatever) and then calculate using the standard formula:

$\hat{p} \pm z_{\frac{\alpha}{2}} \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$

which works pretty well, and then they compare several more complex estimates as well.

If you have R these are available in the binom package (and probably elsewhere)

Answer 2

The link at the bottom provides a good explanation of the "exact" confidence intervals and explains the problems with CI's when sample size is small. Particularly relevant is the following passage:

The term “Exact Confidence Interval” is a bit of a misnomer. Neyman noted [4] that “exact probability statements are impossible in the case of the Binomial Distribution”. This stems from the fact that k, the number of successes in n trials, must be expressed as an integer. Various methods have been suggested as improvements to the Exact CI, including the Wilson Method and the Modified Wilson Method.

So essentially there is no perfect solution. Peter Flom's suggestion may be as good as any.

Binomial CI link