12
$\begingroup$

The maximum likelihood estimator of an exponential distribution $f(x, \lambda) = \lambda e^{-\lambda x}$ is $\lambda_{MLE} = \frac {n} {\sum x_i}$; I know how to derive that by find the derivative of the log likelihood and setting equal to zero.

I then read in an online article that "Unfortunately this estimator is clearly biased since $<\sum_i x_i>$ is indeed $1/\lambda$ but $<1/\sum_i x_i > \neq \lambda$."

Why does $<\sum_i x_i> = 1/\lambda$? If I am correct in deducing the $< >$ operator means expected value, then I thought $E(x_i) = 1/\lambda$ - that is, the expected value of one such $x_i$, is $1/\lambda$, not the sum of all $x_i$'s. And can someone explain the second of the statement and how these two statements demonstrate the MLE is biased?

$\endgroup$
1
  • $\begingroup$ It's a bit tricky to say there's an error when we don't know what $<.>$ means (context might help), but indeed it must be the case that since $\bar x$ is unbiased for $1/\lambda$ that $1/\bar x$ will be biased; it's a consequence of Jensen's inequality. This is not necessarily a problem, most MLEs are biased; it's not clear to me that unbiasedness is especially important (generally I'd tend to think something more like low MSE would be more important than unbiasedness). $\endgroup$
    – Glen_b
    May 31, 2014 at 2:57

1 Answer 1

18
$\begingroup$

I cannot speak as to the use of these symbols but let me show you instead the traditional way, why the mle is biased.

Recall that the exponential distribution is a special case of the General Gamma distribution with two parameters, shape $a$ and rate $b$. The pdf of a Gamma Random Variable is:

$$f_Y (y)= \frac{1}{\Gamma(a) b^a} y^{a-1} e^{-y/b}, \ 0<y<\infty$$

where $\Gamma (.)$ is the gamma function. Alternative parameterisations exist, see for example the wikipedia page.

If you put $a=1$ and $b=1/\lambda$ you arrive at the pdf of the exponential distribution:

$$f_Y(y)=\lambda e^{-\lambda y},0<y<\infty$$

One of the most important properties of a gamma RV is the additivity property, simply put that means that if $X$ is a $\Gamma(a,b)$ RV, $\sum_{i=1}^n X_i$ is also a Gamma RV with $a^{*}=\sum a_i$ and $b^{*}=b$ as before.

Define $Y=\sum X_i$ and as noted above $Y$ is also a Gamma RV with shape parameter equal to $n$, $\sum_{i=1}^n 1 $, that is and rate parameter $1/\lambda$ as $X$ above. Now take the expectation $E[Y^{-1}]$

$$ E\left [ Y^{-1} \right]=\int_0^{\infty}\frac{y^{-1}y^{n-1}\lambda^n}{\Gamma(n)}\times e^{-\lambda y}dy=\int_0^{\infty}\frac{y^{n-2}\lambda^n}{\Gamma(n)}\times e^{-\lambda y}dy$$

Comparing the latter integral with an integral of a Gamma distribution with shape parameter $n-1$ and rate one $1/\lambda$ and using the fact that $\Gamma(n)=(n-1) \times \Gamma(n-1)$ we see that it equals $\frac{\lambda}{n-1}$. Thus

$$E\left[ \hat{\theta} \right]=E\left[ \frac{n}{Y} \right]=n \times E\left[Y^{-1}\right]=\frac{n}{n-1} \lambda$$

which clearly shows that the mle is biased. Note, however, that the mle is consistent. We also know that under some regularity conditions, the mle is asymptotically efficient and normally distributed, with mean the true parameter $\theta$ and variance $\{nI(\theta) \}^{-1} $. It is therefore an optimal estimator.

Does that help?

$\endgroup$
9
  • $\begingroup$ Also note that the MLE is asymptotically unbiased in the limit as $n \rightarrow \infty$. $\endgroup$ May 30, 2014 at 18:40
  • 6
    $\begingroup$ @BrianBorchers, that's what JohnK meant by "consistent." $\endgroup$ May 30, 2014 at 18:45
  • 1
    $\begingroup$ @TheBigAmbiguous yes, but it wasn't clear to me that the original poster would understand that. $\endgroup$ May 30, 2014 at 19:06
  • $\begingroup$ (+1) John, for the thoroughness. I would only suggest that you change notation for the parameters - the $(a,b)$ notation for the gamma distribution is in many corners used for the "shape, rate" parametrization, rather than for the "shape, scale" one - and the wikipedia page you link to, attests to that. $\endgroup$ May 31, 2014 at 21:56
  • 1
    $\begingroup$ @TheBigAmbiguous, JohnK, and BrianBorchers: Asymptotically unbiased and consistent are not synonyms (and neither implies the other). Let's try not to confuse them, for the sake of clarity. JohnK, note also that consistency is not a particularly strong property, so it doesn't say much regarding "optimality" of any type. $\endgroup$
    – cardinal
    Jun 1, 2014 at 12:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.