3
$\begingroup$

This is a pretty basic question, I think, but I'm finding it difficult to locate an answer or decent treatment of it on the web.

For a binary dataset (binomial) with some unknown parameter p (probability of "heads" or success or something observed), how does the accuracy of the estimated p depend on the number of trials?

I've read a number of ways to calculate confidence intervals around the estimated p, but these seem to all depend on assuming a large number of trials and a normal distribution. I'm interested in particular here with small numbers of trials.

Thanks for any solutions or pointers.

$\endgroup$
1

2 Answers 2

3
$\begingroup$

This actually has a large literature. A good entry point is Agresti & Coull. They recommend one simple estimate which is to add 2 failures and two successes (or heads and tails or whatever) and then calculate using the standard formula:

$\hat{p} \pm z_{\frac{\alpha}{2}} \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$

which works pretty well, and then they compare several more complex estimates as well.

If you have R these are available in the binom package (and probably elsewhere)

$\endgroup$
1
  • $\begingroup$ Thanks Peter. I had seen this formula but didn't delve into it as I thought it was dependent on assuming normality. $\endgroup$ Dec 20, 2013 at 19:03
2
$\begingroup$

The link at the bottom provides a good explanation of the "exact" confidence intervals and explains the problems with CI's when sample size is small. Particularly relevant is the following passage:

The term “Exact Confidence Interval” is a bit of a misnomer. Neyman noted [4] that “exact probability statements are impossible in the case of the Binomial Distribution”. This stems from the fact that k, the number of successes in n trials, must be expressed as an integer. Various methods have been suggested as improvements to the Exact CI, including the Wilson Method and the Modified Wilson Method.

So essentially there is no perfect solution. Peter Flom's suggestion may be as good as any.

Binomial CI link

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.