Show that the value is, indeed, the MLE

Question

Let $ X_1, ... X_n$ i.i.d with pdf

$$f(x;\theta)=\frac{x+1}{\theta(\theta+1)}\exp(-x/\theta), x>0, \theta >0$$

It is asked to find the MLE estimator for $\theta.$

The likelihood function is given by

$$L(\theta;x)=[\theta(1-\theta)]^{-n}\exp\left(\frac{\sum_i x_i}{\theta}\right)\prod_i (x_i+1)I_{(0,\infty)}(x_i)$$

Then, the derivative of log-likelihood will be

$$\frac{dlogL(\theta;x)}{d\theta}=\frac{-(2\theta+1)n}{\theta(1+\theta)} + \frac{\sum X_i}{\theta^2}$$

I've obtained my candidate to MLE. But, doing the second derivative of the log-likelihood, I could not conclude that it is negative so that the candidate is, indeed, the point of maximum. What should I do, in this case?

Answer 1

You actually don't have to show that it is a maximum in this case. The root of the first derivative of the log-likelihood, that is the MLE, can be shown to be unique if the iid observation are considered from a random variable of the exponential family. That is, a r.v. for which the density has the form:

$f(x;\theta) = h(x)\,\exp{(s\,\theta - K(\theta) )}$

Where $h$ is a function only of the observations $x_i$, $\theta$ is called the natural parameter, $s$ is called the natural statistics and $K$ is a function only of the natural parameter.

In this case, given $X_1,..X_n$ iid with the density you have, we have

$\prod_{i=1}^n \frac{(x_i+1)}{\theta\,(1+\theta)} \exp(-x_i/\theta) \longrightarrow \prod_i (x_i+1) exp\big(-\frac{n\overline{X}}{\theta} - \log(\theta(1+\theta)) \big)$

As this random variable belongs to the exponential family the MLE is unique.

In general, for exponential families the MLE always exists, is unique, is consistent and is asymptotically normal.

EDIT: http://www.stat.purdue.edu/~dasgupta/ml.pdf is a good explanation of this, maybe a bit too mathematical, but it depends from your academic background. Otherwise, V.S.Huzurbazar, in his "The likelihood equation, consistency and the maxima of the likelihood function" (1947) explains this theory in an easier way.

Answer 2

(The likelihood function in the question is written mistakenly, the derivative of the log-likelihood is correct).

To address the specific issue raised by the OP, we need, as mentioned in the comments, to determine whether the second derivative is negative evaluated at the MLE. The tip here is that checking this does not necessarily require to obtain a closed-form expression for the MLE -in fact, sometimes, even if we have a closed-form expression, it is advisable not to use it, but rather, exploit the first-order conditions in a different way.

And this is exactly the case here. The first-order condition eventually leads to a second-degree polynomial in $\theta$ (including the sample mean of the data as a parameter), with a single admissible root, So we have a closed-form expression for the MLE. Denoting the sample mean by $\bar x$, this expression here is

$$\hat \theta = \frac {\bar x-1 + \sqrt{\bar x^2 +6\bar x +1}}{4} >0 \tag{1}$$

Let's turn now to the issue of the sign of the 2nd derivative. The second derivative is

$$\frac{d^2logL(\theta;x)}{d\theta^2}=-n\left(\frac{2\theta+2\theta^2-(2\theta+1)^2}{\theta^2(1+\theta)^2}\right) - 2\frac{\sum X_i}{\theta^3}$$

$$\Rightarrow \frac 1n \frac{d^2logL(\theta;x)}{d\theta^2}=-\frac 1{\theta^2}\left[\frac{-2\theta^2-2\theta-1}{(1+\theta)^2} + 2\frac{\bar x}{\theta}\right] \tag{2}$$

We want the term inside the brackets in $(2)$ to be positive. Now imagine inserting $(1)$, the closed-form expression for the MLE, into $(2)$, and try to determine the sign of the exrpession...

...Instead, it is more efficient to go back to the first-order condition and eliminate the sample mean, since the MLE must satisfy:

$$\bar x = \frac{(2\hat \theta+1)\hat \theta}{(1+\hat \theta)} \tag{3}$$

Inserting $(3)$ into the bracketed expression in $(2)$ we obtain

$$[\;] = \frac{-2\hat \theta^2-2\hat \theta-1}{(1+\hat \theta)^2} + 2\frac{1}{\hat \theta}\frac{(2\hat \theta+1)\hat \theta}{(1+\hat \theta)} $$

which for sign purposes is equivalent to

$$\frac{-2\hat \theta^2-2\hat \theta-1}{(1+\hat \theta)} + 2(2\hat \theta+1) >0$$

$$\Rightarrow (4\hat \theta+2)(1+\hat \theta) > 2\hat \theta^2+2\hat \theta+1$$

which holds, since $\hat \theta >0$.

Answer 3

Removing the multiplicative constants that do not depend on $\theta$, the likelihood function in this case is:

$$\begin{equation} \begin{aligned} L_\mathbf{x}(\theta) &= \prod_{i=1}^n \frac{1}{\theta (\theta+1)} \cdot \exp \Big( - \frac{x_i}{\theta} \Big) \\[6pt] &= \frac{1}{\theta^n (\theta+1)^n} \cdot \exp \Big( - \frac{n \bar{x}}{\theta} \Big). \\[6pt] \end{aligned} \end{equation}$$

Thus, the log-likelihood function is:

$$\ell_\mathbf{x}(\theta) \equiv \ln L_\mathbf{x}(\theta) = -n \Bigg[ \ln(\theta) + \ln(\theta+1) + \frac{\bar{x}}{\theta} \Bigg] \quad \quad \quad \text{for all } \theta >0.$$

I will write the derivatives of this function out in a form that is useful for finding the critical points and then finding the second derivative at those critical points:

$$\begin{equation} \begin{aligned} \frac{d \ell_\mathbf{x}}{d \theta}(\theta) &= -n \Bigg[ \frac{1}{\theta} + \frac{1}{\theta+1} - \frac{\bar{x}}{\theta^2} \Bigg] \\[6pt] &= -n \Bigg[ \frac{2\theta + 1}{\theta(\theta+1)} - \frac{\bar{x}}{\theta^2} \Bigg] \\[6pt] &= - \frac{n}{\theta^2} \Bigg[ \frac{(2\theta + 1)\theta}{\theta+1} - \bar{x} \Bigg], \\[6pt] \frac{d^2 \ell_\mathbf{x}}{d \theta^2}(\theta) &= -n \Bigg[ - \frac{1}{\theta^2} - \frac{1}{(\theta+1)^2} + \frac{2\bar{x}}{\theta^3} \Bigg] \\[6pt] &= -n \Bigg[ - \frac{(\theta+1)^2 + \theta^2}{\theta^2 (\theta+1)^2} + \frac{2\bar{x}}{\theta^3} \Bigg] \\[6pt] &= -n \Bigg[ - \frac{2 \theta^2 + 2 \theta + 1}{\theta^2 (\theta+1)^2} + \frac{2\bar{x}}{\theta^3} \Bigg] \\[6pt] &= -n \Bigg[ \frac{(2\theta+1)(\theta+1) - \theta}{\theta^2 (\theta+1)^2} - \frac{2\bar{x}}{\theta^3} \Bigg] \\[6pt] &= \frac{n}{\theta^3} \Bigg[ \frac{[(2\theta+1) - \theta]\theta}{\theta+1} - 2\bar{x} \Bigg] \\[6pt] &= \frac{n}{\theta^3} \Bigg[ \frac{(2\theta+1) \theta}{\theta+1} - \bar{x} - \frac{\theta^2}{\theta+1} - \bar{x} \Bigg] \\[6pt] &= - \frac{1}{\theta} \frac{d \ell_\mathbf{x}}{d \theta}(\theta) - \frac{n}{\theta^3} \Bigg( \frac{\theta^2}{\theta+1} + \bar{x} \Bigg). \\[6pt] \end{aligned} \end{equation}$$

From this form, we see that at any critical point we must have:

$$\begin{equation} \begin{aligned} \frac{d^2 \ell_\mathbf{x}}{d \theta^2}(\theta) &= - \frac{n}{\theta^3} \Bigg( \frac{\theta^2}{\theta+1} + \bar{x} \Bigg) < 0. \\[6pt] \end{aligned} \end{equation}$$

Since every critical point is a local maximum, this means that there is a unique critical point that is the MLE of the function. Thus, the MLE is obtained by solving the equation:

$$\bar{x} = \frac{(2 \hat{\theta} + 1) \hat{\theta}}{\hat{\theta} + 1} \quad \quad \quad \implies \quad \quad \quad 2 \hat{\theta}^2 + \hat{\theta} (1-\bar{x}) - \bar{x} = 0.$$

This is a quadratic equation with explicit solution:

$$\hat{\theta} = \frac{1}{4} \Bigg[ \bar{x} - 1 + \sqrt{\bar{x}^2 + 6 \bar{x} + 1} \Bigg].$$