5
$\begingroup$

Consider the following distribution belonging to the exponential family.

$p_{\theta}(x) = \theta e^{-\theta x} $

The MLE estimating the $\theta$ parameter is

$\newcommand{\thetaMLE}{\hat{\theta}_{\mathrm{MLE}}}\thetaMLE = 1/\bar{X}$

where $\bar{X}$ is the sample mean.

How do we find the bias of this estimator?

$\mathrm{bias}(\thetaMLE) = E[\thetaMLE] - \theta = E[1/\bar{X}] - \theta$

How do we find expectation of the reciprocal of the sample mean?

$\endgroup$
  • $\begingroup$ Is this homework? If so, please add the homework tag. $\endgroup$ – cardinal Oct 9 '11 at 1:28
  • $\begingroup$ Hint: $1/\bar{X} = n / S_n$ where $S_n = \sum_{k=1}^n X_k$. Now, what is the distribution of $S_n$? From that, can you find $\mathbb E (1/S_n)$? Use linearity to conclude. $\endgroup$ – cardinal Oct 9 '11 at 1:28
  • $\begingroup$ No this is not homework. Thought of this question while studying for exam. $\endgroup$ – Rohit Banga Oct 9 '11 at 2:24
  • $\begingroup$ This is a pretty standard example/exercise found in most intro math stats books. What text are you using for the class? $\endgroup$ – cardinal Oct 9 '11 at 17:19
  • $\begingroup$ Mostly lecture notes $\endgroup$ – Rohit Banga Oct 9 '11 at 18:42
5
$\begingroup$

First you find the distribution of the sample mean. The easiest way to do this is to use moment generating function. For exponential distribution, we have

$$m_{X_i}(t)=E\left[\exp\left(tX_i\right)\right]=\left(1-\frac{t}{\theta}\right)^{-1}$$

For sample mean we have

$$m_{\overline{X}}(t)=E\left[\exp\left(t \overline{X}\right)\right]=E\left[\exp\left(tN^{-1}\sum_{i=1}^{N}X_i\right)\right]=E\left[\prod_{i=1}^{N}\exp(tN^{-1}X_{i})\right]$$

Because of independence, we can interchange the product and expectation operations. so we get.

$$m_{\overline{X}}(t)=\prod_{i=1}^{N}E\left[\exp(tN^{-1}X_{i})\right]=\prod_{i=1}^{N}m_{X_i}(tN^{-1})=\left(1-\frac{t}{N\theta}\right)^{-N}$$

This is the moment generating function of a gamma distribution with shape parameter $N$ and inverse scale parameter $N\theta$, which has mean value of $\frac{1}{\theta}$, showing that the MLE is unbiased for the parameter $\beta=\frac{1}{\theta}$.

Now we simply take the expected value of $\frac{1}{\overline{X}}$ where $\overline{X}\sim Gamma(N,N\theta)$, which is given by:

$$E\left(\frac{1}{\overline{X}}\right)=\int_0^{\infty}\frac{1}{\overline{X}}f(\overline{X})d\overline{X}=\int_0^{\infty}\frac{1}{\overline{X}}\frac{(N\theta)^N \overline{X}^{N-1}\exp(-N\theta \overline{X})}{\Gamma(N)}d\overline{X}$$

In the integral, make the change of variables $t=N\theta \overline{X}\implies dt=N\theta d\overline{X}$, and we get

$$E\left(\frac{1}{\overline{X}}\right)=\frac{1}{\Gamma(N)}\int_0^{\infty}\frac{N\theta}{t}t^{N-1}\exp(-t)dt$$

$$=\frac{N\theta}{\Gamma(N)}\int_0^{\infty}t^{N-2}\exp(-t)dt=\frac{N\theta\Gamma(N-1)}{\Gamma(N)}=\frac{N\theta}{N-1}$$

Provided that $N\neq 1$, otherwise the expectation does not exist (and hence niether does the bias). So you have a bias of:

$$E\left(\frac{1}{\overline{X}}-\theta\right)=\frac{\theta}{N-1}$$

$\endgroup$
  • $\begingroup$ That seems to be a nice solution. I would not have thought on the lines of mgf and gamma distributions given my noob status. $\endgroup$ – Rohit Banga Oct 9 '11 at 2:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.