I'm refreshing my knowledge of statistics, and I'm stuck on a problem that asks for a comparison between 2 estimators (unfortunately, I can't remember the source of the problem, but I do know its intent is to show that unbiased estimators aren't necessarily the best).


Your colleague wishes to estimate the unknown success probability $p$ of a 2-sided (but possibly unfair) coin. To do so, he flips the coin and observes whether it is heads ($X_1 = 1$) or tails ($X_1 = 0$). If he observes heads, he proceeds to flip the coin an additional $k-1$ times for a total of $k$ flips, and he records the total number of heads obtained in the $k$ flips $\left(\sum_{i=1}^k x_i \right)$. If he observes tails, he does not flip the coin any more and just records the value of $X_1$


$$N = X_1I(X_1=0) + \left(1+\sum_{i=2}^k X_i \right)I(X_1 =1)$$ where $I$ is the indicator function. Compare the mean squared error of two candidate estimators of $p$: (1) $X_1$ or (2) $\frac{N}{k}$ (i.e., the sample proportion).


It is relatively straightforward to see that $X_1$, while unbiased, isn't a very good estimator for $p$ because it will say $p=0$ or $p=1$. However, because it is unbiased, the mean squared error is simply the variance of $X_1$, which is $p(1-p)$ since it is a Bernoulli trial.

To calculate the mean squared error for the 2nd estimator, we need to figure out both its variance and bias. I claim that

\begin{eqnarray} \mathrm{var}\left( \frac{N}{k} \right) &=& \mathrm{var} \left( \mathbb{E}\left( \frac{N}{k} \big| X_1 \right) \right) + \mathbb{E} \left( \mathrm{var} \left( \frac{N}{k} \big| X_1 \right) \right) \\ &=& \frac{p^2}{k^2}(k-1)p(1-p) + \\& &\frac{p-p^2}{k^2} \left(1 + 2 (k-1)p + (k-1)p(1-p) + \left((k-1)p \right)^2 \right) \; \; \text{(*)} \end{eqnarray}

and that the expected value of $\frac{N}{k}$ is

$$\mathbb{E} \left( \frac{N}{k} \right) = \mathbb{E} \left( \mathbb{E} \left( \frac{N}{k} \big| X_1 \right) \right) = \frac{p}{k} \left(1+(k-1)p \right)$$ which means $\frac{N}{k}$ is a biased estimator, with the bias being

$$\mathrm{bias} \left( \frac{N}{k} \right) = \left[\frac{p}{k} (1+(k-1)p) - p \right] \; \; \text{(**)}$$

Therefore, the mean squared error of $\frac{N}{k}$ is the sum of (*) and (**) squared.

Does this seem reasonable? I would have expected the mean squared error of the 2nd estimator $\left( \frac{N}{k} \right)$ to be smaller than the mean squared error of the 1st estimator ($X_1$), thereby showing what I believe is the point of the exercise (i.e., sometimes biased estimators can be more efficient). However, based on what I've done, this doesn't seem to be the case, which makes me think I messed up somewhere (i.e., things don't seem to cancel out, etc. when I try to simplify (*) + (**)$^2$). Any guidance or insights would be appreciated.


Here is how I'm calculating the pieces of (*) in case someone can point on my error

\begin{eqnarray} \mathrm{var}_{X_1} \left( \mathbb{E}_{\frac{N}{k}|X_1}\left( \frac{N}{k} \big| X_1 \right) \right) &=& \mathrm{var}_{X_1} \left(\frac{1}{k}\mathbb{E}_{N | X_1} \left(N | X_1 \right) \right) \nonumber \\ &=& \mathrm{var}_{X_1} \left( \frac{1}{k} \left( 0 \times (1-p) + \left(1+\sum_{i=2}^k x_i\right) \times p \right) \right) \nonumber \\ &=& \frac{1}{k^2}\mathrm{var}_{X_1} \left( p + p\sum_{i=2}^k x_i \right) \nonumber \\ &=& \frac{p^2}{k^2}\mathrm{var}_{X_1} \left(\sum_{i=2}^k x_i \right) \nonumber \\ &=& \frac{p^2}{k^2}(k-1)p(1-p) \end{eqnarray}

\begin{eqnarray} \mathbb{E}_{X_1} \left( \mathrm{var}_{\frac{N}{k}|X_1} \left( \frac{N}{k} \big| X_1 \right) \right) &=& \mathbb{E}_{X_1} \left( \frac{1}{k^2}\mathrm{var}_{N|X_1} \left( N | X_1 \right) \right) \nonumber \\ &=& \mathbb{E}_{X_1} \left( \frac{1}{k^2} \left( \mathbb{E}(N^2 | X_1) - \left( \mathbb{E}(N | X_1 \right)^2 \right) \right) \nonumber \\ &=& \mathbb{E}_{X_1} \left( \frac{1}{k^2} \left( p \left(1+\sum_{i=2}^k x_i \right)^2 - p^2 \left(1 + \sum_{i=2}^k x_i\right)^2 \right) \right) \nonumber \\ &=& \mathbb{E}_{X_1} \left( \frac{1}{k^2} \left( p-p^2 \right) \left(1+\sum_{i=2}^k x_i \right)^2 \right) \nonumber \\ &=& \frac{p-p^2}{k^2}\mathbb{E}_{X_1} \left(1+2\sum_{i=2}^k x_i + \left( \sum_{i=2}^k x_i \right)^2 \right) \\ &=& \frac{p-p^2}{k^2} \left(\mathbb{E}_{X_1}(1) + 2 \mathbb{E}_{X_1} \left(\sum_{i=2}^k x_i \right) + \mathbb{E}_{X_1}\left( \sum_{i=2}^k x_i \right)^2 \right) \\ &=& \frac{p-p^2}{k^2} \left(1 + 2 (k-1)p + \mathrm{var}\left(\sum_{i=2}^k x_i \right) + \left( \mathbb{E}\left(\sum_{i=2}^k x_i \right) \right)^2 \right) \\ &=& \frac{p-p^2}{k^2} \left(1 + 2 (k-1)p + (k-1)p(1-p) + \left((k-1)p \right)^2 \right) \end{eqnarray}

  • $\begingroup$ The third equality of the second derivation does not hold. The expectation of $I(X_1)$ can not be evaluated inside $\mathbb{E}_{X_1}$, and the expectations $\mathbb{E}(N^2|X_1)$ and $\mathbb{E}(N|X_1)$ are not correct. $\endgroup$ – Apeln Mar 12 '15 at 16:42

There is most likely some mistake in your calculation of $Var\left(\frac{N}{k}\right)$. First, note that the term $X_1I(X_1=0)$ is always 0, so it does not need to be considered further. Then,




Because the first term is always 0, we do not have to consider it or the covariance in the derivation of $Var\left(N\big|X_1\right)$. We get

$Var\left(N\big|X_1\right)=Var\left((1+\sum_{i=2}^kX_i)I(X_1=1)\big|X_1\right)=(I(X_1=1))^2 Var\left(1+\sum_{i=2}^kX_i\right)=(I(X_1=1))^2(k-1)p(1-p)$



Finally, we obtain



MSE$\left(\frac{N}{k}\right)=\frac{p(1-p)}{k^2}(1+(k+2)(k-1)p)$ (after simplification).

This yields the following for $k=100$: enter image description here

| cite | improve this answer | |
  • $\begingroup$ Thank you for this. Would you mind editing your answer to show how you derived the variance and expectation above? I'm not quite sure where I'm going wrong, so seeing your work might help. $\endgroup$ – RetiredChemist Mar 11 '15 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.