Is there an unbiased estimator of PMF of a random variable $Y=\sum_{i=1}^{n} X_n $ where $X_i$ are independent Bernoulli trials with probability $p$, that is, the estimator of: \begin{equation}\tag{1} f(k,n)=P(Y=k|n)={n\choose k}p^k(1-p)^{n-k} \end{equation} for an arbitrary $n \geq 0$ and $n\geq k \geq 0$ when based on a sample of $s $ observations from that said Bernoulli RV; $\lbrace x_i\rbrace_{i=1}^{s}$?

More info and progress so far:
An obvious candidate for the estimator of PMF would be: \begin{equation}\tag{2} \hat{f}(k,n)={n\choose k}\hat{p}^k(1-\hat{p})^{n-k} \end{equation} where $\hat{p}= \frac{\sum_{i=1}^{s}x_i}{s}$. That is consistent but clearly not unbiased as apparent from Jensen's inequality. So far I found that the estimator based on hypergeometric distribution: \begin{equation}\tag{3} \hat{f}(k,n)=\dfrac{{\hat{p}*s\choose k} {s-\hat{p}*s\choose n-k}}{{n\choose k}} \end{equation} with $\hat{p}$ defined as above seems to be unbiased (at least in my simulations, although I wasn't able to prove it). This function $\hat{f}(k,n)$ is however usefull only for $n\leq s$. Does there exist some unbiased estimator of $f(k,n)$ which would work also for $n>s$? (Possibly some correction of estimator (2) or some generalization of estimator (3). Or something completely different.)


1 Answer 1


Since, for a Binomial $\text{B}(n,p)$ variable $X$, and $k\le n$, the factorial moment is given by $$\mathbb{E}_p[X(X-1)\cdots(X-k+1)] = n(n-1)\cdots(n-k+1)p^k,$$ the $s$ Bernoulli rvs $\lbrace X_i\rbrace_{i=1}^{s}$ can easily return independent unbiased estimates of both $p^k$ and $(1-p)^{n-k}$ if $k+(n-k)\le s$, that is, if $n\le s$. And hence of $\mathbb{P}(X=k)\propto p^k(1-p)^{n-k}$.

It sounds likely that an unbiased estimator of the above does not exist when $n>s$, because, for instance, developing $(X_1+\ldots+X_s)^k$ shows that the maximum number of terms in a product is $s$, with expectation $p^s$. No higher power of $p$ or $(1-p)$ can appear for this reason. Actually, the proof is straightforward: consider there exists such an unbiased estimator, denoted by $G(X_1+\ldots+X_s)$ since by sufficiency there exists an unbiased estimator based on the sum. Then it satisfies $$\mathbb{E}_p[G(X_1+\ldots+X_s)]=\sum_{j=1}^s \underbrace{G(j){s \choose j}}_\text{independent from $p$}p^j(1-p)^{s-j}=p^k(1-p)^{n-k}$$ or $$\sum_{j=1}^s \overbrace{G(j){s \choose j}}^{\text{non-negative}}p^{j-k}(1-p)^{s-j-n+k}=1$$ Letting $p$ tend to $0$ or $1$ leads to explosive terms when $j-k<0$ and when $s-j-n+k<0$, unless the coefficient $$G(j){s \choose j}p^{j-k}$$ is equal to zero. If $n>s$, then, for all $0\le j\le s$ and all $0\le k\le n$, either $j<k$ or $s-j<n-k$, which leads to an impossibility since $G(m)=0$ for all $0\le m\le s$. Therefore there is no unbiased estimator of $p^k(1-p)^{n-k}$ when $n>s$.

  • 1
    $\begingroup$ Thank you very much, professor Xi'an. That is a very neat counterexample. Now I know that my search for such estimator is futile and can allocate my time accordingly. Thanks again. $\endgroup$
    – Nicolle
    Jan 6, 2019 at 19:07
  • $\begingroup$ Thank you, this is a very neat case about the rarity of functions that allow for an unbiased estimator, a rarity that is often forgotten by textbooks. $\endgroup$
    – Xi'an
    Jan 6, 2019 at 19:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.