8
$\begingroup$

I am new to statistics especially in the topic of estimators and sufficient statistic.

I am reading a note which says "unbiasedness is a desirable (but not necessary) property of a good estimator". Then it gives an example where unbiased estimators don't exist. Since there is no further explanation in the example, I am struggling to understand it.

The example says:

If $X\sim B(\varphi(\theta))$ for some function $\varphi$, then $\theta^*\in K_0\iff\mathbb{E}_\theta\theta^*\equiv\theta^*(0)(1-\varphi(\theta))+\theta^*(1)\varphi(\theta)=\theta,\forall\theta\in\Theta)$, (where $K_0$ is the class of all unbiased estimators).

This is just for reference:
An estimator $\theta_0^*=\theta_0^*(X)$ from a class $K$ of estimators of $\theta$ is called efficient in $K$ if, for any $\theta^*\in K$, $$\mathbb{E_\theta}(\theta_0^*-\theta)^2\le\mathbb{E_\theta}(\theta*-\theta)^2,\forall\theta\in\Theta.$$

For a function $b=b(\theta),\theta\in\Theta$, let $$K_b=\{\theta^*:\mathbb{E}_\theta\theta^*=\theta+b(\theta),\forall\theta\in\Theta\}$$ be the class of all estimators with the bias $b(\theta)$.

I understand the basic idea of estimators, the property for being biased and unbiased. But I have no clue why unbiased estimators don't exist in the given example above. May anyone kindly explain it to me please?

$\endgroup$
1

2 Answers 2

9
$\begingroup$

I interpret "$B(p)$" to mean a Bernoulli distribution with parameter $p = \Pr(X=1)$ and I suppose $X$ is a single observation from this distribution. Trivial though this situation is, it is instructive and even yields a surprising result: the statement in the question is not universally correct.


An estimator $\theta^*$ can be expressed as a vector $(x, y)$ where $x$ is your estimate when you observe $X=0$ and $y$ is your estimate when you observe $X=1.$ These are just numbers.

Writing $p = \phi(\theta),$ the unbiasedness condition -- which must hold for all possible $\theta$ -- is

$$\theta = x(1-p) + yp = x + p(x - y).$$

(The second equality is simple algebra.)

This has two immediate, simple consequences:

  1. $\theta$ and $p$ must be linearly related if an unbiased estimator exists.

  2. When they are linearly related, which means $\theta = a + bp,$ then $x$ and $y$ are uniquely determined by $x = a$ and $y = a + b.$ In this case, unbiased estimators do exist.

A common, natural example of (2) is where $\theta = p = 0 + 1(p).$ We must use $x = 0$ and $y = 1,$ whence the estimator is

$$\theta^*(X) = 0 + 1(X) = X.$$

That's intuitively right: when our estimate is $\theta^* = \hat p = 1$ when we observe a coin land heads and otherwise our estimate is $\hat p = 0,$ the expectation of that estimate is precisely $p:$ it's unbiased (even though the individual values of any estimate are extreme).

Another revealing example is where the space of possible values of $p$ is restricted. Suppose, for instance, that $p\in\{1/3, 2/3\}$ and you wish to estimate $\theta = \log p.$ Then, although this looks nonlinear, it is not, because (on this restricted domain),

$$\theta(p) = \log(1/3) + \frac{\log(2/3) - \log(1/3)}{2/3 - 1/3}(p - 1/3) = a + bp$$

indeed is linear and an unbiased estimator exists for this family with $x=a$ and $y = a+b.$

Conversely, this analysis yields a rich set of examples of situations with no unbiased estimators -- and that must be the point of the quoted passage. Letting $p$ be any value between $0$ and $1$ (as usual), for instance, we conclude there are no unbiased estimators of any polynomials of degree $2$ or greater, no unbiased estimators of $\log p,$ and so on.


You can read about non-trivial extensions of these ideas to Binomial sampling (that is, larger samples of Bernoulli variables) by searching our site.

$\endgroup$
5
$\begingroup$

$\bullet$ The go-to way to check unbiasedness of, say, $\psi(\theta), $ when the family $\{f(\mathbf x;\theta),~\theta\in\Theta\}$ is a power series distribution, is to equate the coefficients of $\theta^i, ~i=1, 2,\ldots, $ when $\psi(\theta) $ is an analytic function (cf. $\rm [I]$).

$\bullet$ (The notations used suggest that the note is probably based on $\rm[II]$. I would stick to the author's notations rather than in OP.) Consider a Bernoulli scheme with unknown $p:=\Pr[x_1=1].$ Suppose we are seeking an unbiased estimator of $\varphi(p), ~\varphi$ a measurable function. Then a possible estimator $g(\mathbf x) $ would be of the form $$\sum_\mathbf xg(\mathbf x) f(\mathbf x; p) = \sum_{k=0}^n G(k) p^k(1-p) ^{n-k}=\varphi(p), \tag 1\label 1$$ where $G(k) :=\sum_{\left\{\mathbf x :\sum_i \mathbf I(x_i=1)~=~k\right\}}g(\mathbf x).$ It is evident from $\eqref 1$ that it can be solvable only if $\varphi(p) $ is a polynomial of degree at most $n.$


$\rm[III]$ shows when a functional $F(\mathbf P), ~\mathbf P\in\mathfrak D\subset \mathfrak D^*,~\mathfrak D^*$ being the set of all probability distributions on $\mathbb R, $ admits an unbiased estimator:

A necessary and sufficient condition that $F$ have an unbiased estimate of order $n$ over $\mathfrak D$ is that it be homogenous over $\mathfrak D$ of degree $k\leq n. $

$F$ is homogeneous over $\mathfrak D$ of degree $k$ if there exists a real-valued measurable function $\varphi:=\varphi(x_1, \ldots, x_k) $ such that $$\int\cdots\int \varphi(x_1, \ldots, x_k) ~\mathrm d\mathbf P(x_1) \cdots\mathrm d\mathbf P(x_k) = F(\mathbf P), ~~\forall ~\mathbf P\in\mathfrak D, $$ and $k$ is minimal for such representation.

The result, however, doesn't ascertain any satisfactory estimators. For that additional features like symmetry is needed, which the author explained subsequently.


References:

$\rm [I]$ A First Course in Parametric Inference, B. K. Kale, Narosa Publishing House, $1999,$ sec. $3.1, $ p. $46.$

$\rm[II]$ Mathematical Statistics, A. A. Borovkov, OPA, $1998, $ sec. $18,$ p. $93.$

$\rm [III]$ The Theory of Unbiased Estimation, Paul. R. Halmos, Ann. Math. Statist. $17(1):~ 34-43$ (March, $1946$). DOI: $10.1214$/aoms/$1177731020.$

$\endgroup$
1
  • 2
    $\begingroup$ +1 especially for the "exotic" references :-) $\endgroup$
    – utobi
    Jan 10 at 8:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.