Exponential distribution: Log-Likelihood and Maximum Likelihood estimator

Question

For a random variable with its CDF given by $$F(x)=1-\exp(-\lambda x),$$ and its PDF given by $$f(x)=\lambda \exp(-\lambda x),$$ for $x>0$ and $\lambda >0$.

How would I write the log-likelihood function for a random sample $X_1,X_2,...,X_n$ i.i.d. Exp($\lambda$) and a maximum likelihood estimator for $\lambda$?

I know that the exponential distribution is . I'm not quite sure where to go from there. Any help would be appreciated.

Answer 1

The likelihood is given as

$$L(\lambda,x) = L(\lambda,x_1,...,x_N) = \prod_{i=1}^N f(x_i,\lambda)$$

where the second identity use the IID assumption and with $x = (x_1,...,x_N)$. The log-likelikelihood is given as

$$l(\lambda,x) := log L(\lambda,x) = \sum_{i=1}^N \log f(x_i, \lambda),$$

where $log f(x_i,\lambda) = log \lambda - \lambda x_i$. This implies that

$$l(\lambda,x) = \sum_{i=1}^N log \lambda - \lambda x_i = N \log \lambda - \lambda \sum_{i=1}^N x_i.$$ Since we are interested in maximum a positive monotone transformation such as dividing with $N$ is fine. This gets us to

$$\frac{1}{N} l(\lambda , x) = \log \lambda - \lambda \bar x$$

differentiate and set to zero to get first order condition

$$\frac{1}{\lambda} - \bar x = 0 \Leftrightarrow \lambda = \frac{1}{\bar x}$$

Small simulation in R

lambda_0 <- 0.5
N <- 10000
x <- rexp(N, rate = lambda_0)

loglik <- function(theta)
    {
        ll <- N * log(theta) - theta*sum(x)
        return(ll)
    }

# Calculate estimate
m_x <- 1/mean(x)

# Create vector for plot of loglikelihood
t <- seq(0.5*m_x,1.5*m_x,length.out=100)

plot(t,loglik(t),type="l")
abline(v=m_x,col="red")

This will genrate this plot of loglikelihood function to see maximum ...

Answer 2

Any standard mathematical statistics book would give you detailed approach of how to do log likelihood. If you want better understanding of Likelihood theory then I would recommend a wonderful text In all Likelihood by Pawitan.

For your specific problem Likelihood $L$ can be written as :

$$f(\mathbf{x},\beta) = \frac{1}{\beta} \ e^{\left(\frac{-\mathbf{x}}{\beta}\right)}; \mathbf{x}>0$$

$$L(\beta,\mathbf{x}) = L(\beta,x_1,...,x_N) = \prod_{i=1}^N f(x_i,\beta)$$

$$L(\beta,\mathbf{x}) = \prod_{i=1}^N \frac{1}{\beta} \ e^{\left(\frac{-x_i}{\beta}\right)} $$

The Log likelihood $\mathscr{L} = log(L)$:

$$\mathscr{L}(\beta,\mathbf{x}) = log\left(\prod_{i=1}^N \frac{1}{\beta} \ e^{\left(\frac{-x_i}{\beta}\right)} \right)$$

So doing some algebra and applying properties of Logarithms you get:

$$\mathscr{L}(\beta,\mathbf{x}) = log\left(\prod_{i=1}^N \frac{1}{\beta} \ e^{\left(\frac{-x_i}{\beta}\right)}\right) = \sum_{i=1}^N \left( log\left(\frac{1}{\beta}\right) + log\left( e^{\left(\frac{-x_i}{\beta}\right)} \right) \right)$$

Since the first part of equation has nothing to do with summation take $log(\frac{1}{\beta})$ outside of summation.

$$\mathscr{L}(\beta,\mathbf{x}) = N \ log\left(\frac{1}{\beta}\right) + \sum_{i=1}^N \left( \frac{- x_i} {\beta} \right)$$

The above can be further simplified:

$$\mathscr{L}(\beta,\mathbf{x}) = - N \ log(\beta) + \frac{1}{\beta}\sum_{i=1}^N -x_i$$

To get the maximum likelihood, take the first partial derivative with respect to $\beta$ and equate to zero and solve for $\beta$:

$$ \frac{\partial \mathscr{L}}{\partial \beta} = \frac{\partial}{\partial \beta} \left(- N \ log(\beta) + \frac{1}{\beta}\sum_{i=1}^N -x_i \right) = 0$$

$$ \frac{\partial \mathscr{L}}{\partial \beta} = -\frac{N} {\beta} + \frac{1} {\beta^2} \sum_{i=1}^N x_i = 0$$

Now solving for $\beta$ you get:

$$\boxed{\beta = \frac{\sum_{i=1}^N x_i}{N} = \overline{\mathbf{x}}}$$

Please note that in your question $\lambda$ is parameterized as $\frac {1} {\beta}$ in the exponential distribution. Regardless of parameterization, the maximum likelihood estimator should be the same.