# Maximum likelihood estimate: Is this possible to solve?

I have the following problem:

Formulate the likelihood function, the log-likelihood function, and the maximum-likelihood estimate as well as the Fisher information and the observed Fisher information for each of the following problems.

c) $X_i,\dots,X_n \overset{\mathrm{i.i.d}}{\sim} \mathcal{N}(\theta,\theta^2)$

d) $X_i,\dots,X_n \overset{\mathrm{i.i.d}}{\sim} \mathcal{N}(\theta,\theta)$

I have the following so far:

First, did I do the right things so far? And more important: How can I calculate the maximum likelihood estimate? Or is this even possible?

EDIT

After CoolSerdash's hint I got the following:

Is that correct? But I have some doubts to formulate the observed Fisher information. Wouldn't this term get a beast inserting the maximum-likelihood estimate?

Similar question

I have to do the same for a Gamma distribution: $X_i,\dots,X_n \overset{\mathrm{i.i.d}}{\sim} \Gamma(\alpha,\beta)$ where $\alpha$ is known.

I have the following so far:

Does this make sense?

• They way you write it, $S(\theta)$ is not the maximum likelihood estimate, but the first derivative of the log-likelihood w.r.t. the unknown parameter. And what do we do with the first derivative in order to find a maximum? ... you should end up with a quadratic polynomial in $\theta$. Oct 6 '13 at 22:21
• Thanks! I know that... but I was not able to transform $S(\theta)$ so that I get a solution for $\theta_{ML}$. Do you maybe have a hint? Oct 6 '13 at 22:29
• You can derive the maximum likelihood estimator by setting the gradient vector (also known as the score) to equal zero and then solve for $\theta$. To calculate the maximum likelihood estimate, you'll need some observed data. Oct 6 '13 at 22:32
• Take $\frac {1}{\theta^3}$ out of the sum in c) as common factor and $\frac {1}{\theta^2}$ out of the sum in d). It does not affect the finding of zero of $S(\theta)$ anymore, does it? And you are left with quadratic polynomials in each case. Oct 6 '13 at 22:37
• Thanks Alecos. But that is my problem. How do I solve $\sum \left( x_i(x_i - \theta) - \theta^2\right) = 0$? And will there be a quite easy solution? Sorry I am a biologist taking an advanced stats. class and I am not very familiar with sums... Oct 7 '13 at 7:13

The second problem (d), where the mean is equal to the variance is discussed on pp. 53 of Asymptotic Theory of Statistics and Probability by Anirban DasGupta (2008). The $\mathcal{N}(\theta, \theta)$ distribution, the normal distribution with an equal mean and variance can be seen as a continuous analog of the Poisson distribution.

I will try to outline a path to the solutions.

The log-likelihood function of a $\mathcal{N}(\mu, \sigma^{2})$ is given by:

$$\ell(\mu, \sigma^2)=-\frac{n}{2}\log(2\pi)-\frac{n}{2}\log(\sigma^2)-\frac{1}{2\sigma^{2}}\sum_{i=1}^{n}(x_{i}-\mu)^{2}.$$ Setting $\mu=\sigma^{2}=\theta$ yields $$\ell(\theta)=-\frac{n}{2}\log(2\pi)-\frac{n}{2}\log(\theta)-\frac{1}{2\theta}\sum_{i=1}^{n}(x_{i}-\theta)^{2}.$$ Expanding the term under the sum leads to \begin{align} \ell(\theta) &=-\frac{n}{2}\log(2\pi)-\frac{n}{2}\log(\theta)-\frac{1}{2\theta}\left(\sum_{i=1}^{n}x_{i}^{2}-2\theta\sum_{i=1}^{n}x_{i}+n\theta^{2}\right) \\ &=-\frac{n}{2}\log(2\pi)-\frac{n}{2}\log(\theta)-\frac{s}{2\theta}+t-\frac{n\theta}{2} \\ \end{align} where $s=\sum_{i=1}^{n}x_{i}^{2}$ and $t=\sum_{i=1}^{n}x_{i}$. Taking the first derivative wrt $\theta$ gives $$S(\theta)=\frac{d}{d\theta}\ell(\theta)=\frac{s}{2\theta^{2}}-\frac{n}{2\theta}-\frac{n}{2}.$$ So $s$ is the minimal sufficient statistic. The maximum likelihood estimator $\hat{\theta}$ can be found by setting $S(\theta)=0$ and solving for $\theta$.

Applying the same procedure to $\mathcal{N}(\mu=\theta, \sigma^{2}=\theta^{2})$, the log-likelihood function is $$\ell(\theta)=-\frac{n}{2}\log(2\pi)-\frac{n}{2}\log(\theta^{2})-\frac{1}{2\theta^{2}}\sum_{i=1}^{n}(x_{i}-\theta)^{2}.$$ This leads to the following score function (again, with $s=\sum_{i=1}^{n}x_{i}^{2}$ and $t=\sum_{i=1}^{n}x_{i}$): $$S(\theta)=\frac{s}{\theta^{3}}-\frac{t}{\theta^2}-\frac{n}{\theta}.$$

## Fisher information

The Fisher information is defined as the negative second derivative of the log-likelihood function: $$I(\theta)=-\frac{d^{2}\,\ell(\theta)}{d\,\theta^{2}}=-\frac{d\,S(\theta)}{d\,\theta}.$$ The observed Fisher information is $I(\hat{\theta})$, the Fisher information evaluated at the maximum likelihood estimate.

For the second question (d), we have: $$I(\theta)=-\frac{d}{d\,\theta}\left(\frac{s}{2\theta^{2}}-\frac{n}{2\theta}-\frac{n}{2} \right) = \frac{s}{\theta^{3}}-\frac{n}{2\theta^{2}}.$$

And for the first question (c), we have: $$I(\theta)=-\frac{d}{d\,\theta}\left(\frac{s}{\theta^{3}}-\frac{t}{\theta^2}-\frac{n}{\theta}\right) = \frac{3s}{\theta^{4}}-\frac{2t}{\theta^{3}}-\frac{n}{\theta^{2}}.$$ To get the observed Fisher information, plug in the maximum likelihood estimates.

## Gamma distribution

It looks right to me but you don't need the sums in the expressions of the Fisher information.