4
$\begingroup$

Let $ X_1, ... X_n$ i.i.d with pdf

$$f(x;\theta)=\frac{x+1}{\theta(\theta+1)}\exp(-x/\theta), x>0, \theta >0$$

It is asked to find the MLE estimator for $\theta.$

The likelihood function is given by

$$L(\theta;x)=[\theta(1-\theta)]^{-n}\exp\left(\frac{\sum_i x_i}{\theta}\right)\prod_i (x_i+1)I_{(0,\infty)}(x_i)$$

Then, the derivative of log-likelihood will be

$$\frac{dlogL(\theta;x)}{d\theta}=\frac{-(2\theta+1)n}{\theta(1+\theta)} + \frac{\sum X_i}{\theta^2}$$

I've obtained my candidate to MLE. But, doing the second derivative of the log-likelihood, I could not conclude that it is negative so that the candidate is, indeed, the point of maximum. What should I do, in this case?

$\endgroup$
  • $\begingroup$ How did a factor of $(1-\theta)^{-n}$ appear in the likelihood when $(1+\theta)^{-1}$ appears in $f$ (as it must, to normalize it)? Why did the sign change? Why is the negative sign missing in the exponential? (Those look like typos; your final equation seems correct.) $\endgroup$ – whuber Sep 18 '14 at 21:14
  • $\begingroup$ Hint: You don't have to show that the second derivative is negative everywhere; you only want to evaluate it at the optimum value of $\theta$. $\endgroup$ – whuber Sep 18 '14 at 21:22
  • $\begingroup$ What do you mean by "optimum value" of theta ? $\endgroup$ – Giiovanna Sep 18 '14 at 21:52
  • 2
    $\begingroup$ That's the entire point: the sign of the second derivative is relevant only at $\hat\theta$; for determining whether $\hat\theta$ is a (local) maximum, the sign is not relevant anywhere else. I don't know what you mean by "not have a good form", but I believe that if you work through the algebra, you will find it's a particularly simple rational function of a minimal sufficient statistic $\sum X_i/n$ and the parameter $\theta$ and it can be expressed in a way that is obviously negative. $\endgroup$ – whuber Sep 18 '14 at 22:29
  • 1
    $\begingroup$ Ok, forget about the form. You are right, I was having a problem with calculus here. Thank you very much for your attention. $\endgroup$ – Giiovanna Sep 18 '14 at 22:34
1
$\begingroup$

You actually don't have to show that it is a maximum in this case. The root of the first derivative of the log-likelihood, that is the MLE, can be shown to be unique if the iid observation are considered from a random variable of the exponential family. That is, a r.v. for which the density has the form:

$f(x;\theta) = h(x)\,\exp{(s\,\theta - K(\theta) )}$

Where $h$ is a function only of the observations $x_i$, $\theta$ is called the natural parameter, $s$ is called the natural statistics and $K$ is a function only of the natural parameter.

In this case, given $X_1,..X_n$ iid with the density you have, we have

$\prod_{i=1}^n \frac{(x_i+1)}{\theta\,(1+\theta)} \exp(-x_i/\theta) \longrightarrow \prod_i (x_i+1) exp\big(-\frac{n\overline{X}}{\theta} - \log(\theta(1+\theta)) \big)$

As this random variable belongs to the exponential family the MLE is unique.

In general, for exponential families the MLE always exists, is unique, is consistent and is asymptotically normal.

EDIT: http://www.stat.purdue.edu/~dasgupta/ml.pdf is a good explanation of this, maybe a bit too mathematical, but it depends from your academic background. Otherwise, V.S.Huzurbazar, in his "The likelihood equation, consistency and the maxima of the likelihood function" (1947) explains this theory in an easier way.

$\endgroup$
  • $\begingroup$ These are useful observations, thank you. But: (1) You will need to generalize your definition of exponential family slightly in order to make your point--or else change the parameter from $\theta$ to $1/\theta$--because the density in question is not exactly in the characteristic form you give. It would also be of great help to any reader new to these concepts to be clear at the outset that $s$ is a function of $x$ alone. (2) The OP has found the derivative but observes the second derivative is not everywhere negative. It doesn't look like your observation directly addresses that confusion. $\endgroup$ – whuber Sep 19 '14 at 15:29
1
$\begingroup$

(The likelihood function in the question is written mistakenly, the derivative of the log-likelihood is correct).

To address the specific issue raised by the OP, we need, as mentioned in the comments, to determine whether the second derivative is negative evaluated at the MLE. The tip here is that checking this does not necessarily require to obtain a closed-form expression for the MLE -in fact, sometimes, even if we have a closed-form expression, it is advisable not to use it, but rather, exploit the first-order conditions in a different way.

And this is exactly the case here. The first-order condition eventually leads to a second-degree polynomial in $\theta$ (including the sample mean of the data as a parameter), with a single admissible root, So we have a closed-form expression for the MLE. Denoting the sample mean by $\bar x$, this expression here is

$$\hat \theta = \frac {\bar x-1 + \sqrt{\bar x^2 +6\bar x +1}}{4} >0 \tag{1}$$

Let's turn now to the issue of the sign of the 2nd derivative. The second derivative is

$$\frac{d^2logL(\theta;x)}{d\theta^2}=-n\left(\frac{2\theta+2\theta^2-(2\theta+1)^2}{\theta^2(1+\theta)^2}\right) - 2\frac{\sum X_i}{\theta^3}$$

$$\Rightarrow \frac 1n \frac{d^2logL(\theta;x)}{d\theta^2}=-\frac 1{\theta^2}\left[\frac{-2\theta^2-2\theta-1}{(1+\theta)^2} + 2\frac{\bar x}{\theta}\right] \tag{2}$$

We want the term inside the brackets in $(2)$ to be positive. Now imagine inserting $(1)$, the closed-form expression for the MLE, into $(2)$, and try to determine the sign of the exrpession...

...Instead, it is more efficient to go back to the first-order condition and eliminate the sample mean, since the MLE must satisfy:

$$\bar x = \frac{(2\hat \theta+1)\hat \theta}{(1+\hat \theta)} \tag{3}$$

Inserting $(3)$ into the bracketed expression in $(2)$ we obtain

$$[\;] = \frac{-2\hat \theta^2-2\hat \theta-1}{(1+\hat \theta)^2} + 2\frac{1}{\hat \theta}\frac{(2\hat \theta+1)\hat \theta}{(1+\hat \theta)} $$

which for sign purposes is equivalent to

$$\frac{-2\hat \theta^2-2\hat \theta-1}{(1+\hat \theta)} + 2(2\hat \theta+1) >0$$

$$\Rightarrow (4\hat \theta+2)(1+\hat \theta) > 2\hat \theta^2+2\hat \theta+1$$

which holds, since $\hat \theta >0$.

$\endgroup$
1
$\begingroup$

Removing the multiplicative constants that do not depend on $\theta$, the likelihood function in this case is:

$$\begin{equation} \begin{aligned} L_\mathbf{x}(\theta) &= \prod_{i=1}^n \frac{1}{\theta (\theta+1)} \cdot \exp \Big( - \frac{x_i}{\theta} \Big) \\[6pt] &= \frac{1}{\theta^n (\theta+1)^n} \cdot \exp \Big( - \frac{n \bar{x}}{\theta} \Big). \\[6pt] \end{aligned} \end{equation}$$

Thus, the log-likelihood function is:

$$\ell_\mathbf{x}(\theta) \equiv \ln L_\mathbf{x}(\theta) = -n \Bigg[ \ln(\theta) + \ln(\theta+1) + \frac{\bar{x}}{\theta} \Bigg] \quad \quad \quad \text{for all } \theta >0.$$

I will write the derivatives of this function out in a form that is useful for finding the critical points and then finding the second derivative at those critical points:

$$\begin{equation} \begin{aligned} \frac{d \ell_\mathbf{x}}{d \theta}(\theta) &= -n \Bigg[ \frac{1}{\theta} + \frac{1}{\theta+1} - \frac{\bar{x}}{\theta^2} \Bigg] \\[6pt] &= -n \Bigg[ \frac{2\theta + 1}{\theta(\theta+1)} - \frac{\bar{x}}{\theta^2} \Bigg] \\[6pt] &= - \frac{n}{\theta^2} \Bigg[ \frac{(2\theta + 1)\theta}{\theta+1} - \bar{x} \Bigg], \\[6pt] \frac{d^2 \ell_\mathbf{x}}{d \theta^2}(\theta) &= -n \Bigg[ - \frac{1}{\theta^2} - \frac{1}{(\theta+1)^2} + \frac{2\bar{x}}{\theta^3} \Bigg] \\[6pt] &= -n \Bigg[ - \frac{(\theta+1)^2 + \theta^2}{\theta^2 (\theta+1)^2} + \frac{2\bar{x}}{\theta^3} \Bigg] \\[6pt] &= -n \Bigg[ - \frac{2 \theta^2 + 2 \theta + 1}{\theta^2 (\theta+1)^2} + \frac{2\bar{x}}{\theta^3} \Bigg] \\[6pt] &= -n \Bigg[ \frac{(2\theta+1)(\theta+1) - \theta}{\theta^2 (\theta+1)^2} - \frac{2\bar{x}}{\theta^3} \Bigg] \\[6pt] &= \frac{n}{\theta^3} \Bigg[ \frac{[(2\theta+1) - \theta]\theta}{\theta+1} - 2\bar{x} \Bigg] \\[6pt] &= \frac{n}{\theta^3} \Bigg[ \frac{(2\theta+1) \theta}{\theta+1} - \bar{x} - \frac{\theta^2}{\theta+1} - \bar{x} \Bigg] \\[6pt] &= - \frac{1}{\theta} \frac{d \ell_\mathbf{x}}{d \theta}(\theta) - \frac{n}{\theta^3} \Bigg( \frac{\theta^2}{\theta+1} + \bar{x} \Bigg). \\[6pt] \end{aligned} \end{equation}$$

From this form, we see that at any critical point we must have:

$$\begin{equation} \begin{aligned} \frac{d^2 \ell_\mathbf{x}}{d \theta^2}(\theta) &= - \frac{n}{\theta^3} \Bigg( \frac{\theta^2}{\theta+1} + \bar{x} \Bigg) < 0. \\[6pt] \end{aligned} \end{equation}$$

Since every critical point is a local maximum, this means that there is a unique critical point that is the MLE of the function. Thus, the MLE is obtained by solving the equation:

$$\bar{x} = \frac{(2 \hat{\theta} + 1) \hat{\theta}}{\hat{\theta} + 1} \quad \quad \quad \implies \quad \quad \quad 2 \hat{\theta}^2 + \hat{\theta} (1-\bar{x}) - \bar{x} = 0.$$

This is a quadratic equation with explicit solution:

$$\hat{\theta} = \frac{1}{4} \Bigg[ \bar{x} - 1 + \sqrt{\bar{x}^2 + 6 \bar{x} + 1} \Bigg].$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.