Trying to understand an example where unbiased estimators don't exist

Question

I am new to statistics especially in the topic of estimators and sufficient statistic.

I am reading a note which says "unbiasedness is a desirable (but not necessary) property of a good estimator". Then it gives an example where unbiased estimators don't exist. Since there is no further explanation in the example, I am struggling to understand it.

The example says:

If $X\sim B(\varphi(\theta))$ for some function $\varphi$, then $\theta^*\in K_0\iff\mathbb{E}_\theta\theta^*\equiv\theta^*(0)(1-\varphi(\theta))+\theta^*(1)\varphi(\theta)=\theta,\forall\theta\in\Theta)$, (where $K_0$ is the class of all unbiased estimators).

This is just for reference:
An estimator $\theta_0^*=\theta_0^*(X)$ from a class $K$ of estimators of $\theta$ is called efficient in $K$ if, for any $\theta^*\in K$, $$\mathbb{E_\theta}(\theta_0^*-\theta)^2\le\mathbb{E_\theta}(\theta*-\theta)^2,\forall\theta\in\Theta.$$

For a function $b=b(\theta),\theta\in\Theta$, let $$K_b=\{\theta^*:\mathbb{E}_\theta\theta^*=\theta+b(\theta),\forall\theta\in\Theta\}$$ be the class of all estimators with the bias $b(\theta)$.

I understand the basic idea of estimators, the property for being biased and unbiased. But I have no clue why unbiased estimators don't exist in the given example above. May anyone kindly explain it to me please?

Answer 1

I interpret "$B(p)$" to mean a Bernoulli distribution with parameter $p = \Pr(X=1)$ and I suppose $X$ is a single observation from this distribution. Trivial though this situation is, it is instructive and even yields a surprising result: the statement in the question is not universally correct.

---

An estimator $\theta^*$ can be expressed as a vector $(x, y)$ where $x$ is your estimate when you observe $X=0$ and $y$ is your estimate when you observe $X=1.$ These are just numbers.

Writing $p = \phi(\theta),$ the unbiasedness condition -- which must hold for all possible $\theta$ -- is

$$\theta = x(1-p) + yp = x + p(x - y).$$

(The second equality is simple algebra.)

This has two immediate, simple consequences:

1. $\theta$ and $p$ must be linearly related if an unbiased estimator exists.

2. When they are linearly related, which means $\theta = a + bp,$ then $x$ and $y$ are uniquely determined by $x = a$ and $y = a + b.$ In this case, unbiased estimators do exist.

A common, natural example of (2) is where $\theta = p = 0 + 1(p).$ We must use $x = 0$ and $y = 1,$ whence the estimator is

$$\theta^*(X) = 0 + 1(X) = X.$$

That's intuitively right: when our estimate is $\theta^* = \hat p = 1$ when we observe a coin land heads and otherwise our estimate is $\hat p = 0,$ the expectation of that estimate is precisely $p:$ it's unbiased (even though the individual values of any estimate are extreme).

Another revealing example is where the space of possible values of $p$ is restricted. Suppose, for instance, that $p\in\{1/3, 2/3\}$ and you wish to estimate $\theta = \log p.$ Then, although this looks nonlinear, it is not, because (on this restricted domain),

$$\theta(p) = \log(1/3) + \frac{\log(2/3) - \log(1/3)}{2/3 - 1/3}(p - 1/3) = a + bp$$

indeed is linear and an unbiased estimator exists for this family with $x=a$ and $y = a+b.$

Conversely, this analysis yields a rich set of examples of situations with no unbiased estimators -- and that must be the point of the quoted passage. Letting $p$ be any value between $0$ and $1$ (as usual), for instance, we conclude there are no unbiased estimators of any polynomials of degree $2$ or greater, no unbiased estimators of $\log p,$ and so on.

---

You can read about non-trivial extensions of these ideas to Binomial sampling (that is, larger samples of Bernoulli variables) by searching our site.

Answer 2

$\bullet$ The go-to way to check unbiasedness of, say, $\psi(\theta), $ when the family $\{f(\mathbf x;\theta),~\theta\in\Theta\}$ is a power series distribution, is to equate the coefficients of $\theta^i, ~i=1, 2,\ldots, $ when $\psi(\theta) $ is an analytic function (cf. $\rm [I]$).

$\bullet$ (The notations used suggest that the note is probably based on $\rm[II]$. I would stick to the author's notations rather than in OP.) Consider a Bernoulli scheme with unknown $p:=\Pr[x_1=1].$ Suppose we are seeking an unbiased estimator of $\varphi(p), ~\varphi$ a measurable function. Then a possible estimator $g(\mathbf x) $ would be of the form $$\sum_\mathbf xg(\mathbf x) f(\mathbf x; p) = \sum_{k=0}^n G(k) p^k(1-p) ^{n-k}=\varphi(p), \tag 1\label 1$$ where $G(k) :=\sum_{\left\{\mathbf x :\sum_i \mathbf I(x_i=1)~=~k\right\}}g(\mathbf x).$ It is evident from $\eqref 1$ that it can be solvable only if $\varphi(p) $ is a polynomial of degree at most $n.$

---

$\rm[III]$ shows when a functional $F(\mathbf P), ~\mathbf P\in\mathfrak D\subset \mathfrak D^*,~\mathfrak D^*$ being the set of all probability distributions on $\mathbb R, $ admits an unbiased estimator:

A necessary and sufficient condition that $F$ have an unbiased estimate of order $n$ over $\mathfrak D$ is that it be homogenous over $\mathfrak D$ of degree $k\leq n. $

$F$ is homogeneous over $\mathfrak D$ of degree $k$ if there exists a real-valued measurable function $\varphi:=\varphi(x_1, \ldots, x_k) $ such that $$\int\cdots\int \varphi(x_1, \ldots, x_k) ~\mathrm d\mathbf P(x_1) \cdots\mathrm d\mathbf P(x_k) = F(\mathbf P), ~~\forall ~\mathbf P\in\mathfrak D, $$ and $k$ is minimal for such representation.

The result, however, doesn't ascertain any satisfactory estimators. For that additional features like symmetry is needed, which the author explained subsequently.

---

References:

$\rm [I]$ A First Course in Parametric Inference, B. K. Kale, Narosa Publishing House, $1999,$ sec. $3.1, $ p. $46.$

$\rm[II]$ Mathematical Statistics, A. A. Borovkov, OPA, $1998, $ sec. $18,$ p. $93.$

$\rm [III]$ The Theory of Unbiased Estimation, Paul. R. Halmos, Ann. Math. Statist. $17(1):~ 34-43$ (March, $1946$). DOI: $10.1214$/aoms/$1177731020.$