# Trying to understand an example where unbiased estimators don't exist

I am new to statistics especially in the topic of estimators and sufficient statistic.

I am reading a note which says "unbiasedness is a desirable (but not necessary) property of a good estimator". Then it gives an example where unbiased estimators don't exist. Since there is no further explanation in the example, I am struggling to understand it.

The example says:

If $$X\sim B(\varphi(\theta))$$ for some function $$\varphi$$, then $$\theta^*\in K_0\iff\mathbb{E}_\theta\theta^*\equiv\theta^*(0)(1-\varphi(\theta))+\theta^*(1)\varphi(\theta)=\theta,\forall\theta\in\Theta)$$, (where $$K_0$$ is the class of all unbiased estimators).

This is just for reference:
An estimator $$\theta_0^*=\theta_0^*(X)$$ from a class $$K$$ of estimators of $$\theta$$ is called efficient in $$K$$ if, for any $$\theta^*\in K$$, $$\mathbb{E_\theta}(\theta_0^*-\theta)^2\le\mathbb{E_\theta}(\theta*-\theta)^2,\forall\theta\in\Theta.$$

For a function $$b=b(\theta),\theta\in\Theta$$, let $$K_b=\{\theta^*:\mathbb{E}_\theta\theta^*=\theta+b(\theta),\forall\theta\in\Theta\}$$ be the class of all estimators with the bias $$b(\theta)$$.

I understand the basic idea of estimators, the property for being biased and unbiased. But I have no clue why unbiased estimators don't exist in the given example above. May anyone kindly explain it to me please?

## 2 Answers

I interpret "$$B(p)$$" to mean a Bernoulli distribution with parameter $$p = \Pr(X=1)$$ and I suppose $$X$$ is a single observation from this distribution. Trivial though this situation is, it is instructive and even yields a surprising result: the statement in the question is not universally correct.

An estimator $$\theta^*$$ can be expressed as a vector $$(x, y)$$ where $$x$$ is your estimate when you observe $$X=0$$ and $$y$$ is your estimate when you observe $$X=1.$$ These are just numbers.

Writing $$p = \phi(\theta),$$ the unbiasedness condition -- which must hold for all possible $$\theta$$ -- is

$$\theta = x(1-p) + yp = x + p(x - y).$$

(The second equality is simple algebra.)

This has two immediate, simple consequences:

1. $$\theta$$ and $$p$$ must be linearly related if an unbiased estimator exists.

2. When they are linearly related, which means $$\theta = a + bp,$$ then $$x$$ and $$y$$ are uniquely determined by $$x = a$$ and $$y = a + b.$$ In this case, unbiased estimators do exist.

A common, natural example of (2) is where $$\theta = p = 0 + 1(p).$$ We must use $$x = 0$$ and $$y = 1,$$ whence the estimator is

$$\theta^*(X) = 0 + 1(X) = X.$$

That's intuitively right: when our estimate is $$\theta^* = \hat p = 1$$ when we observe a coin land heads and otherwise our estimate is $$\hat p = 0,$$ the expectation of that estimate is precisely $$p:$$ it's unbiased (even though the individual values of any estimate are extreme).

Another revealing example is where the space of possible values of $$p$$ is restricted. Suppose, for instance, that $$p\in\{1/3, 2/3\}$$ and you wish to estimate $$\theta = \log p.$$ Then, although this looks nonlinear, it is not, because (on this restricted domain),

$$\theta(p) = \log(1/3) + \frac{\log(2/3) - \log(1/3)}{2/3 - 1/3}(p - 1/3) = a + bp$$

indeed is linear and an unbiased estimator exists for this family with $$x=a$$ and $$y = a+b.$$

Conversely, this analysis yields a rich set of examples of situations with no unbiased estimators -- and that must be the point of the quoted passage. Letting $$p$$ be any value between $$0$$ and $$1$$ (as usual), for instance, we conclude there are no unbiased estimators of any polynomials of degree $$2$$ or greater, no unbiased estimators of $$\log p,$$ and so on.

You can read about non-trivial extensions of these ideas to Binomial sampling (that is, larger samples of Bernoulli variables) by searching our site.

$$\bullet$$ The go-to way to check unbiasedness of, say, $$\psi(\theta),$$ when the family $$\{f(\mathbf x;\theta),~\theta\in\Theta\}$$ is a power series distribution, is to equate the coefficients of $$\theta^i, ~i=1, 2,\ldots,$$ when $$\psi(\theta)$$ is an analytic function (cf. $$\rm [I]$$).

$$\bullet$$ (The notations used suggest that the note is probably based on $$\rm[II]$$. I would stick to the author's notations rather than in OP.) Consider a Bernoulli scheme with unknown $$p:=\Pr[x_1=1].$$ Suppose we are seeking an unbiased estimator of $$\varphi(p), ~\varphi$$ a measurable function. Then a possible estimator $$g(\mathbf x)$$ would be of the form $$\sum_\mathbf xg(\mathbf x) f(\mathbf x; p) = \sum_{k=0}^n G(k) p^k(1-p) ^{n-k}=\varphi(p), \tag 1\label 1$$ where $$G(k) :=\sum_{\left\{\mathbf x :\sum_i \mathbf I(x_i=1)~=~k\right\}}g(\mathbf x).$$ It is evident from $$\eqref 1$$ that it can be solvable only if $$\varphi(p)$$ is a polynomial of degree at most $$n.$$

$$\rm[III]$$ shows when a functional $$F(\mathbf P), ~\mathbf P\in\mathfrak D\subset \mathfrak D^*,~\mathfrak D^*$$ being the set of all probability distributions on $$\mathbb R,$$ admits an unbiased estimator:

A necessary and sufficient condition that $$F$$ have an unbiased estimate of order $$n$$ over $$\mathfrak D$$ is that it be homogenous over $$\mathfrak D$$ of degree $$k\leq n.$$

$$F$$ is homogeneous over $$\mathfrak D$$ of degree $$k$$ if there exists a real-valued measurable function $$\varphi:=\varphi(x_1, \ldots, x_k)$$ such that $$\int\cdots\int \varphi(x_1, \ldots, x_k) ~\mathrm d\mathbf P(x_1) \cdots\mathrm d\mathbf P(x_k) = F(\mathbf P), ~~\forall ~\mathbf P\in\mathfrak D,$$ and $$k$$ is minimal for such representation.

The result, however, doesn't ascertain any satisfactory estimators. For that additional features like symmetry is needed, which the author explained subsequently.

## References:

$$\rm [I]$$ A First Course in Parametric Inference, B. K. Kale, Narosa Publishing House, $$1999,$$ sec. $$3.1,$$ p. $$46.$$

$$\rm[II]$$ Mathematical Statistics, A. A. Borovkov, OPA, $$1998,$$ sec. $$18,$$ p. $$93.$$

$$\rm [III]$$ The Theory of Unbiased Estimation, Paul. R. Halmos, Ann. Math. Statist. $$17(1):~ 34-43$$ (March, $$1946$$). DOI: $$10.1214$$/aoms/$$1177731020.$$

• +1 especially for the "exotic" references :-) Commented Jan 10 at 8:08