How to find a good estimator for $\lambda$ in exponential distibution? I have an Exponential distribution with $\lambda$ as a parameter.
How can I find a good estimator for lambda?
 A: The term how to find a good estimator is quite broad. Often we assume an underlying distribution and put forth the claim that data follows the given distribution. We then aim at fitting the distribution on our data. In this case ensuring we minimize the distance (KL-Divergence) between our data and the assumed distribution. This gives rise to Maximum Likelihood Estimation. We thus aim to obtain a parameter which will maximize the likelihood.
In your case, the MLE for $X\sim Exp(\lambda)$ can be derived as:
$$
\begin{aligned}
l(\lambda) =& \sum\log(f(x_i))\quad\text{where} \quad f(x_i)=\lambda e^{-\lambda x}\\
=&n\log\lambda-\lambda\sum x\\
\frac{\partial l(\lambda)}{\partial \lambda} = &\frac{n}{\lambda} - \sum x \quad
\text{setting this to } 0 \text{ and solving for the stationary point}\\
\implies \hat\lambda =& \frac{n}{\sum x} = \frac{1}{\bar x}\end{aligned}
$$
This estimator can be considered as good. But what exactly do we consider as a good estimator? Some properties for a good estimator are:

*

*Unbiasedness - Is our estimator Unbiased?
An estimator $\hat\theta$ will be considered unbiased when $E(\hat\theta) = \theta$
In Our case:
$$
\begin{aligned}
E(\hat\lambda) = & E\left(\frac{1}{\bar X}\right) = E\left(\frac{n}{\sum X_i}\right)= E\left(\frac{n}{y}\right)\\
Recall:\quad& \sum X_i = y \sim \Gamma(\alpha=n, \beta = \lambda) \text{ where } \beta\text{ is the rate parameter}\\
\therefore E\left(\frac{n}{y}\right) = &\int_0^\infty \frac{n}{y}\frac{\lambda^n}{\Gamma(n)}y^{n-1}e^{-\lambda y}dy = n\int_0^\infty \frac{\lambda^n}{\Gamma(n)}y^{n-1-1}e^{-\lambda y}dy = n\frac{\lambda^n}{\Gamma(n)}\frac{\Gamma(n-1)}{\lambda^{n-1}}\\
=&\frac{n}{n-1}\lambda\\
\implies& E\left(\frac{n-1}{n}\hat\lambda\right) = \lambda
\end{aligned}
$$
Our estimator above is biased. But we can have a unbiased estimator $\frac{n-1}{n\bar X}$. There are many other unbiased estimators you could find. But which one is the best? We then look at the notion of Efficiency.


*

*Efficiency
For an exponential random variable,
$$
\ln f(x \mid \lambda)=\ln \lambda-\lambda x, \quad \frac{\partial^{2} f(x \mid \lambda)}{\partial \lambda^{2}}=-\frac{1}{\lambda^{2}}
$$
Thus,
$$
I(\lambda)=\frac{1}{\lambda^{2}}
$$
Now, $\bar{X}$ is an unbiased estimator for $h(\lambda)=1 / \lambda$ with variance
$$
\frac{1}{n \lambda^{2}}
$$
By the Cramér-Rao lower bound, we have that
$$
\frac{g^{\prime}(\lambda)^{2}}{n I(\lambda)}=\frac{1 / \lambda^{4}}{n \lambda^{2}}=\frac{1}{n \lambda^{2}}
$$
Because $\bar X$ attains the lower bound, we say that it is efficient.
You could also look at Consistency, Asymptotic Normality and even Robustness.
Lastly, you would like to look at the MSE of your estimator. In this case:
$$
\begin{aligned}
MSE(\hat\lambda) =&E(\hat\lambda - \lambda)^2 = E(\hat\lambda^2) - 2\lambda E(\hat\lambda) + \lambda^2\\
=&\frac{n^2\lambda^2}{(n-1)(n-2)} -\frac{2n\lambda^2}{n-1}+\lambda^2\\
=&\frac{\lambda^2(n+2)}{(n-1)(n-2)}
\end{aligned}
$$
In the end you will still have to find a balance between the biasedness and MSE. Often a times we aim at reducing both. But usually no one estimator completely minimizes both.
A: The estimation section of @StephanKolassa's Wikipedia link
has the information you need. I illustrate some of the statements
there, using a simulation in R. I use $n = 10$ and $\lambda = 1/3.$
The MLE of $\mu = 1/\lambda$ is $\hat\mu = \bar X$ and it is unbiased:
$E(\hat \mu) = E(\bar X) = \mu.$
The MLE of $\lambda$ is $\hat\lambda = 1/\bar X.$ It is biased (unbiassedness
does not 'survive' a nonlinear transformation): $E[(\hat\lambda-\lambda)] = \lambda/(n-1).$ Thus an unbiased estimator of $\lambda$ based on the MLE
is $\hat\lambda_u = \frac{n-2}{n-1}\frac{1}{\bar X}.$
set.seed(1119)
a = replicate(10^6, mean(rexp(10,1/3)))
mean(a)
[1] 3.000249    # sample mean unbiased MLE for pop mean
mean(1/a)
[1] 0.3702984   # MLE for rate is biased (too large)
lam.est.u = (8/9)*(1/a)
mean(lam.est.u)
[1] 0.3291541   # unbiased MLE for rate; aprx 1/3

The Wikipedia link suggests that the (slightly biased) estimator
$\hat\lambda_m = \frac{n-2}{n}\frac{1}{\bar X}$ may have slightly
better properties.
