Error in the fraction estimated from a finite sample? This seems like a basic statistics question, but I (a non-statistician) have been unable to find a definitive answer in online searches, possibly because I'm not using the right search terms.
Given a very large number of black balls and white balls, you want to estimate the true fraction $f$ of black balls from a small sample.   You take $M$ balls, with replacement, and you find that $N$ of that sample are black.  The naive estimate of the desired true fraction is therefore $\hat{f} = N/M$.   But what is the uncertainty or standard error $\sigma_f$ in $\hat f$?  Ideally, I'm looking for an expression or error  distribution that is valid even for small or zero $N$, but I'll settle for one that's approximately correct for $N \ge 5$.
I recognize that under some conditions the above estimate $\hat f$ can be biased.  For example, if I draw only three balls and none is black, the true fraction can be significantly greater than zero but it cannot be less than zero; i.e., $\hat f = 0/3 = 0$ would seem to be biased low relative to most possibilities for the true fraction.
For a rigorous derivation, it does seem to me that one must make the a priori assumption that all true fractions on the interval [0,1] are equally likely, but beyond that, I'm stuck.
 A: You seem to be thinking of the wrong things as fixed and varying. Which is natural. The definition of unbiased involves fixing $f$ and looking at how $\hat f$ varies. Suppose you have the simplest case with $N=1$ and, say, $f=1/2021$.  The possible results $M$ are

*

*0: probability $2020/2021$

*1: probability $1/2021$
The expected value of $M/N$ is $(0/1)\times P(M=0)+ (1/1)\times P(M=1)$, which is $0\times 2020/2021+1\times 1/2021=1/2021$, which is exactly $f$.
The same calculation can be made for any other $f$
The possible results $M$ are

*

*0: probability $1-f$

*1: probability $f$
The expected value of $M/N$ is $(0/1)\times (1-f)+ (1/1)\times f$, which is $0\times (1-f) +1\times f=f$.
Note what this isn't saying. It isn't talking about $f$ being random or uncertaint or unknown. The lack of bias is true for any fixed $f$, whether you know $f$ or not.
It also isn't saying that $\hat f$ will be above $f$ as often as it is below $f$. That's not the definition of bias; that would be saying the median of $\hat f$ is $f$, which isn't true. The median of $\hat f$ is zero if $f<0.5$, one if $f>0.5$, and up for annoying definitional argument if $f=0.5$.
If you did have some prior probability distribution over $f$ then $E[f|\hat f]$ would make sense and typically $E[f|\hat f]\neq \hat f$.  But it would still be true that $E[\hat f|f]=f$ and $E[\hat f]=E[f]$
A: Since I have not received any answers to my question, I decided to try some numerical experiments in which I generated a large set (20 million realizations) of simulated measurements, with $f$ ranging from 0 to 1 in increments of 0.01, M ranging from 1 to 200, and 1000 randomly generated samples based on the Python/Numpy statement
N = np.sum(np.random.random(M)< f)
The values of the true $f$ were averaged to an $M\times N$ array, and the standard deviation was computed in another array.
What I found is that, for $N > 3$ and $N \ll M$, a reasonably unbiased estimate of the true $f$ is given by
$$
\hat f \approx \frac{N+1}{M}
$$
and the error is
$$
\sigma_f = \frac{\sqrt{N+1}}{M}
$$
The above formulas appear to be correct to within better than 10% of the true value for $N$ as small as 3-4.
Barring a computational error on my part, these results seem good enough to serve my purposes, but I would still welcome a theoretical justification if one exists.
EDIT:  Subsequent to posting the above, I discovered that there is extensive discussion of this problem here and here.  The short version is that there is no single right answer to how to estimate either the fraction $f$ or the confidence interval on $f$ (and none of the formally derived formulas are the same as my empirical expressions above), but the method/formula chosen is subject to various considerations (e.g., the range of $M$ and $N$) and tradeoffs in simplicity and accuracy.
There is also a python program that calculates the confidence interval here.
