# Estimation of probability of a success in binomial distribution

Let's say we have two biased coins. The probability of tossing a head on the first coin is $\alpha$ and the probability of tossing a head on the second coin is $1-\alpha$. We toss both coins $n$ times and we say that there is a success when there is a head on both coins. If we denote this random variable by $X$ then

$$X\sim B(n,\alpha-\alpha^2).$$

The question is how to properly estimate $\alpha$. The problem with a 'standard' approach ($\alpha - \alpha^2=\frac{k}{n}$ where $k$ is number of successes) is that the number of successes might be greater than $0.25n$ and in such a case we obtain that $\alpha$ is a complex number. We can treat such cases as $\alpha=0.5$, but then this estimator is not unbiased anymore.

• You would probably want to treat k/n>0.25 cases as $\alpha=0.5$ and not 0, but that does not change the bias issue. Jan 4, 2011 at 19:17
• @Aniko: Of course You are right, I've already changed it. Jan 4, 2011 at 19:21

The estimator is biased, regardless.

Note first that $\alpha$ is not identifiable because you cannot distinguish between $\alpha$ and $1-\alpha$. Let's accommodate this problem by allowing that we don't care which coin is which and stipulating (arbitrarily, but with no loss of generality), that $0 \le \alpha \le 1/2$.

It's reasonable, and conventional, to fix the estimator $g$ as follows:

\eqalign{ g(k,n) =& \frac{1 - \sqrt{\delta}}{2} \cr \delta =& \max(0, 1 - 4 k / n) }

No matter what you do, though, this will be a nonlinear function of the outcome $k$ and therefore is certain to be biased for almost all $\alpha$.

A better approach is to search for estimators $h(k,n)$ among some functional class of estimators (such as ones that are linear in $k$) that minimize the expectation of some loss function. In many situations an estimator that works well for quadratic loss also works well for many reasonable types of losses, so let's look at this. What we're talking about, then, is (for each $n$) to minimize the expectation $\mathbb{E}[(h(k,n) - \alpha)^2]$ among all estimators $h$.

Let's look graphically at what's going on. The bias of any estimator $h$ of the parameter $\alpha$ is the difference between its expectation and the parameter, $\mathbb{E}[h(k,n) - \alpha]$. We can study any proposed estimator, then, by graphing its bias (if we really care about that) and its loss. For any value of $n$ they are functions of $\alpha$, which (of course) is unknown. That's why we have to look at the entire graph.

Here are the bias (blue, dashed) and square root of the expected quadratic loss (red) for $g$ when $n=16$:

(I use the root of the loss because this is directly comparable to the bias.) For example, $g$ is unbiased for $\alpha \approx 1/3$ but otherwise is biased, with the size of the bias largest for $\alpha = 1/2$. The root expected loss is roughly between 0.15 and 0.2 provided $\alpha$ exceeds $1/6$, approximately.

As an alternative, consider linear estimators $h_\lambda(k,n)$ of the form $h_\lambda(k,n) = \lambda(n) k/n$. Here is a plot of $h_2$ also for $n=16$ (but please note the change in scale on the vertical axis):

For most $\alpha$ its bias exceeds that of $g$, but for some $\alpha$ (near 0.4) it actually has less bias. For a wide range of $\alpha$, though, its root expected loss is less than that of $g$. Provided $\alpha \gt 1/5$ or so, this simple estimator clearly outperforms the "obvious" one!

This is not necessarily "the best" linear estimator, however. To illustrate, here is a plot of $h_{4/3}$:

It outperforms both $g$ and $h_2$ for $1/8 \lt \alpha \lt 3/8$, approximately. Note, though, that $g$ outperforms the $h_{\lambda}$ for sufficiently small $\alpha$.

These considerations suggest there is value in knowing something about what $\alpha$ might be: that will tell you which portions of the loss graphs to focus on in selecting among alternative estimators. If, in addition, you have a prior distribution for $\alpha$ you can compute the expected loss (this is now a single number) and use that to compare estimators: your task becomes one of finding an estimator with lowest possible expected loss. This, of course, is a Bayesian estimator.

Regardless, using plots of expected loss is a standard and effective way to compare estimators and to choose ones that are appropriate for any particular problem.

• I've got two questions (at least for now): 1) What is the value of estimator g when n=16 and k=8? I'm asking because it seems to be complex. 2) Could You elaborate $h_{\lambda}(k,n)=\lambda(n)k/n$? I'm not sure what $\lambda(n)$ is. Jan 4, 2011 at 21:20
• @Tomek Good catch: I simply left out the most crucial part of the definition of $\delta$! It's in now. It is the obvious one: when the argument of the square root is negative, just set it to zero. In other words, estimate $\alpha = 1/2$ whenever $n/4 \le k$. (There was no such error in the plots, however: this was only a typo.) $\lambda$ is simply a number but it can change from one $n$ to another. For a fixed $n$ that makes $h$ a linear function of $k$.
– whuber
Jan 4, 2011 at 21:47
• @Whuber: Great, now I know how to make the calculations that You have made, just to clarify $h_{\lambda}=\lambda(n)k/n$ should be changed to $h_{\lambda}=min(0.5,\lambda(n)k/n)$. Thank You very much for the answer, I'm still considering whether something better might be done. Jan 4, 2011 at 23:16
• @Tomek Yes, good observation. In fact, the $h$ I ended up using is $\min(1/2, \lambda k / n, \lambda (n-k) /n)$. Although this is not precisely linear in $k$, it is linear in a "constrained" sense. Linearity doesn't really matter; what matters is to find an effective estimator of any form.
– whuber
Jan 5, 2011 at 1:42
• I've asked one of my professors and he said that he would use Bayes estimator: $p = \frac{k+1}{n+2}\cdot \frac{\beta_{k+2,n-k+1}(0.25)}{\beta_{k+1,n-k+1}(0.25)}$ , where $\beta$ denotes CDF of beta distribution and $p=\alpha-\alpha^2$. p is less than 0.25 so it can be easily transformed to $\alpha$. I havent checked those calculations yet, but several simulations, that I've made, have shown that for small n this is far from being great. Jan 13, 2011 at 22:04