This seems like a basic statistics question, but I (a non-statistician) have been unable to find a definitive answer in online searches, possibly because I'm not using the right search terms.

Given a very large number of black balls and white balls, you want to estimate the true fraction $f$ of black balls from a small sample. You take $M$ balls, with replacement, and you find that $N$ of that sample are black. The naive estimate of the desired true fraction is therefore $\hat{f} = N/M$. But what is the uncertainty or standard error $\sigma_f$ in $\hat f$? Ideally, I'm looking for an expression or error distribution that is valid even for small or zero $N$, but I'll settle for one that's approximately correct for $N \ge 5$.

I recognize that under some conditions the above estimate $\hat f$ can be biased. For example, if I draw only three balls and none is black, the true fraction can be significantly greater than zero but it cannot be less than zero; i.e., $\hat f = 0/3 = 0$ would seem to be biased low relative to most possibilities for the true fraction.

For a rigorous derivation, it does seem to me that one must make the a priori assumption that all true fractions on the interval [0,1] are equally likely, but beyond that, I'm stuck.

  • 1
    $\begingroup$ The most useful search term (at least here on CV) is "Clopper," which will direct you to threads discussing accurate small-sample Binomial confidence intervals, most of which mention the Clopper-Pearson method. $\endgroup$
    – whuber
    Feb 18 at 20:49

You seem to be thinking of the wrong things as fixed and varying. Which is natural. The definition of unbiased involves fixing $f$ and looking at how $\hat f$ varies. Suppose you have the simplest case with $N=1$ and, say, $f=1/2021$. The possible results $M$ are

  • 0: probability $2020/2021$
  • 1: probability $1/2021$

The expected value of $M/N$ is $(0/1)\times P(M=0)+ (1/1)\times P(M=1)$, which is $0\times 2020/2021+1\times 1/2021=1/2021$, which is exactly $f$.

The same calculation can be made for any other $f$ The possible results $M$ are

  • 0: probability $1-f$
  • 1: probability $f$

The expected value of $M/N$ is $(0/1)\times (1-f)+ (1/1)\times f$, which is $0\times (1-f) +1\times f=f$.

Note what this isn't saying. It isn't talking about $f$ being random or uncertaint or unknown. The lack of bias is true for any fixed $f$, whether you know $f$ or not.

It also isn't saying that $\hat f$ will be above $f$ as often as it is below $f$. That's not the definition of bias; that would be saying the median of $\hat f$ is $f$, which isn't true. The median of $\hat f$ is zero if $f<0.5$, one if $f>0.5$, and up for annoying definitional argument if $f=0.5$.

If you did have some prior probability distribution over $f$ then $E[f|\hat f]$ would make sense and typically $E[f|\hat f]\neq \hat f$. But it would still be true that $E[\hat f|f]=f$ and $E[\hat f]=E[f]$

  • $\begingroup$ If I'm understanding your comment correctly, my use of "biased" may be incorrect in this context. The more important issue for me, and the essence of my question is, what is the uncertainty in my estimate of the true f? Correct me if I'm wrong, but I don't think that's covered in your response. $\endgroup$ Feb 16 at 4:11

Since I have not received any answers to my question, I decided to try some numerical experiments in which I generated a large set (20 million realizations) of simulated measurements, with $f$ ranging from 0 to 1 in increments of 0.01, M ranging from 1 to 200, and 1000 randomly generated samples based on the Python/Numpy statement

N = np.sum(np.random.random(M)< f)

The values of the true $f$ were averaged to an $M\times N$ array, and the standard deviation was computed in another array.

What I found is that, for $N > 3$ and $N \ll M$, a reasonably unbiased estimate of the true $f$ is given by $$ \hat f \approx \frac{N+1}{M} $$ and the error is $$ \sigma_f = \frac{\sqrt{N+1}}{M} $$ The above formulas appear to be correct to within better than 10% of the true value for $N$ as small as 3-4.

Barring a computational error on my part, these results seem good enough to serve my purposes, but I would still welcome a theoretical justification if one exists.

EDIT: Subsequent to posting the above, I discovered that there is extensive discussion of this problem here and here. The short version is that there is no single right answer to how to estimate either the fraction $f$ or the confidence interval on $f$ (and none of the formally derived formulas are the same as my empirical expressions above), but the method/formula chosen is subject to various considerations (e.g., the range of $M$ and $N$) and tradeoffs in simplicity and accuracy.

There is also a python program that calculates the confidence interval here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.