Let $x_1, x_2... x_n$ be a random sample from a distribution with pdf:

$$f(x;\mu,\sigma)=\frac1{\sigma}\exp\left({-\frac{x-\mu}{\sigma}}\right)\,,-\infty<\mu<\infty;\, \sigma>0;\, x\ge\mu$$

How do I find the MLE for the parameters if both parameters are unknown?

I tried using the usual MLE with likelihood function:

$$L(\mu,\sigma|x_1...x_n)=\frac{1}{\sigma^n}\exp\left({-\frac{\sum{x_i}-n\mu}{\sigma}}\right)$$ But the derivative of this with respect of $\mu$ is a dead end.

I do know that if $\sigma$ is known, the MLE for $\mu$ is $\frac{\sum{x_i}}{n}$ and if $\mu$ is known, the MLE for $\sigma$ is $\frac{\sum{x_i}-n\mu}{n}$.

Do these help? What should be the approach?

  • 2
    $\begingroup$ 1. You should state the limits on the variable and the parameters; those are part of the definition of the density, and an important part of the reasoning here. 2. Not every optimization problem is solved by setting a derivative to 0. Start with a simpler problem by setting $\sigma=1$, choosing an explicit sample (e.g. 1.13, 1.56, 2.08) and draw the log-likelihood function. The required logic should be obvious $\endgroup$ – Glen_b Aug 20 '17 at 12:23
  • $\begingroup$ There's additional clarification and hints for the simplified problem here $\endgroup$ – Glen_b Aug 20 '17 at 12:36
  • $\begingroup$ I have provided the limits. Yes, I am aware that not all optimization can be solved using derivatives. That was how I got the MLE of $\mu$ when $\sigma$ is constant. However, I am having some difficulty on doing the same for when 2 variables $(\mu, \sigma)$ are considered. I will try the approach you stated. $\endgroup$ – user164144 Aug 20 '17 at 13:00
  • $\begingroup$ When $\sigma=1$, I arrive at the conclusion that $\hat{\mu}=\bar{x}$ which I got already. I think this is the MLE for $\mu$ regardless of the value of $\sigma$ based on eyeballing the likelihood. However, that still leaves me without an estimate for $\sigma$ $\endgroup$ – user164144 Aug 20 '17 at 13:10
  • 2
    $\begingroup$ I'm sorry but no, if you set $\sigma=1$ then $\hat{\mu}$ is not $\bar{x}$. If you set $\mu=0$ then $\hat{\sigma}$ would be $\bar{x}$. What did your log-likelihood look like for the specific example? When you sort that out, then try it for a known $\sigma=\sigma_0$. $\endgroup$ – Glen_b Aug 20 '17 at 21:34

Given the sample, the likelihood function is given by $$L(\mu,\sigma)=\frac{1}{\sigma^n}\exp\left[-\frac{1}{\sigma}\sum_{i=1}^n(x_i-\mu)\right]\mathbf1_{\mu\leqslant x_{(1)},\sigma>0}$$

This function is not differentiable at $\mu=x_{(1)}$, so that MLE of $\mu$ has to be found using a different argument. For fixed $\sigma$, $L(\mu,\sigma)$ is an increasing function of $\mu$ $\,\forall\,\sigma$, implying that $\hat\mu_{\text{MLE}}=X_{(1)}$.

MLE of $\sigma$ can be guessed from the first partial derivative as usual.

We have $\displaystyle\frac{\partial L(\mu,\sigma)}{\partial\sigma}=0\implies\sigma=\frac{1}{n}\sum_{i=1}^n(x_i-\mu)$.

So MLE of $\sigma$ could possibly be $\displaystyle\hat\sigma_{\text{MLE}}=\frac{1}{n}\sum_{i=1}^n(X_i-\hat\mu)=\frac{1}{n}\sum_{i=1}^n\left(X_i-X_{(1)}\right)$

The second partial derivative test fails here due to $L(\mu,\sigma)$ not being totally differentiable.

So to confirm that $(\hat\mu,\hat\sigma)$ is the MLE of $(\mu,\sigma)$, one has to verify that $L(\hat\mu,\hat\sigma)\geqslant L(\mu,\sigma)$, or somehow conclude that $\ln L(\hat\mu,\hat\sigma)\geqslant \ln L(\mu,\sigma)$ holds $\forall\,(\mu,\sigma)$.

| cite | improve this answer | |
  • $\begingroup$ Would be interesting to know reason of downvote. $\endgroup$ – StubbornAtom Apr 18 '19 at 6:13

Note that for each fixed $\sigma > 0$, the likelihood $L(\mu,\sigma)$ is an increasing function of $\mu$, provided that $\mu\leq x_{(1)}$ ($x_{(1)}$ being the smallest value of $x$). If $\mu> x_{(1)}$, the likelihood is $0$. Consequently, the MLE of $\mu$ is $\hat{\mu}=x_{(1)}$. Here is a plot of the log-likelihood for a specific example als @Glen_b suggested in the comments ($\sigma = 1, x = \{1.13, 1.56, 2.08\}$):


As for the MLE of $\sigma$, take the first derivative of the log-likelihood, set it to zero and solve for $\sigma$

\begin{align} \log L(x;\mu,\sigma) &=-n\log{(\sigma)}-\frac{1}{\sigma}\sum_{i=1}^{n}{(x_i-\mu)} \\ \frac{\partial \log L(x;\hat{\mu},\sigma)}{\partial \sigma}&= \frac{-n}{\sigma}+\frac{1}{\sigma^2}\sum_{i=1}^{n}{(x_i-\hat{\mu})}\\ \frac{-n}{\sigma}+\frac{1}{\sigma^2}\sum_{i=1}^{n}{(x_i-\hat{\mu})} &= 0\\ \hat{\sigma} &=\frac{1}{n}\sum_{i=1}^{n}(x_i-x_{(1)}) = \bar{x}-x_{(1)} \end{align} where $\bar{x}$ is the sample mean.

These results can be found in the following references.

  • Rahman M & Pearson LM (2001): Estimation in two-parameter exponential distributions. Journal of Statistical Computation and Simulation, 70(4), 371-386.
  • Krishnamoorthy K (2016): Handbook of Statistical Distributions with Applications. 2nd ed. Chapman and Hall/CRC.
| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.