I have the following problem:

Formulate the likelihood function, the log-likelihood function, and the maximum-likelihood estimate as well as the Fisher information and the observed Fisher information for each of the following problems.

c) $X_i,\dots,X_n \overset{\mathrm{i.i.d}}{\sim} \mathcal{N}(\theta,\theta^2)$

d) $X_i,\dots,X_n \overset{\mathrm{i.i.d}}{\sim} \mathcal{N}(\theta,\theta)$

I have the following so far: enter image description here

First, did I do the right things so far? And more important: How can I calculate the maximum likelihood estimate? Or is this even possible?


After CoolSerdash's hint I got the following: enter image description here

Is that correct? But I have some doubts to formulate the observed Fisher information. Wouldn't this term get a beast inserting the maximum-likelihood estimate?

Similar question

I have to do the same for a Gamma distribution: $X_i,\dots,X_n \overset{\mathrm{i.i.d}}{\sim} \Gamma(\alpha,\beta)$ where $\alpha$ is known.

I have the following so far:

enter image description here

Does this make sense?

  • 2
    $\begingroup$ They way you write it, $S(\theta)$ is not the maximum likelihood estimate, but the first derivative of the log-likelihood w.r.t. the unknown parameter. And what do we do with the first derivative in order to find a maximum? ... you should end up with a quadratic polynomial in $\theta$. $\endgroup$ Oct 6 '13 at 22:21
  • $\begingroup$ Thanks! I know that... but I was not able to transform $S(\theta)$ so that I get a solution for $\theta_{ML}$. Do you maybe have a hint? $\endgroup$
    – Michael
    Oct 6 '13 at 22:29
  • $\begingroup$ You can derive the maximum likelihood estimator by setting the gradient vector (also known as the score) to equal zero and then solve for $\theta$. To calculate the maximum likelihood estimate, you'll need some observed data. $\endgroup$ Oct 6 '13 at 22:32
  • 1
    $\begingroup$ Take $\frac {1}{\theta^3}$ out of the sum in c) as common factor and $\frac {1}{\theta^2}$ out of the sum in d). It does not affect the finding of zero of $S(\theta)$ anymore, does it? And you are left with quadratic polynomials in each case. $\endgroup$ Oct 6 '13 at 22:37
  • $\begingroup$ Thanks Alecos. But that is my problem. How do I solve $\sum \left( x_i(x_i - \theta) - \theta^2\right) = 0$? And will there be a quite easy solution? Sorry I am a biologist taking an advanced stats. class and I am not very familiar with sums... $\endgroup$
    – Michael
    Oct 7 '13 at 7:13

The second problem (d), where the mean is equal to the variance is discussed on pp. 53 of Asymptotic Theory of Statistics and Probability by Anirban DasGupta (2008). The $\mathcal{N}(\theta, \theta)$ distribution, the normal distribution with an equal mean and variance can be seen as a continuous analog of the Poisson distribution.

I will try to outline a path to the solutions.

The log-likelihood function of a $\mathcal{N}(\mu, \sigma^{2})$ is given by:

$$ \ell(\mu, \sigma^2)=-\frac{n}{2}\log(2\pi)-\frac{n}{2}\log(\sigma^2)-\frac{1}{2\sigma^{2}}\sum_{i=1}^{n}(x_{i}-\mu)^{2}. $$ Setting $\mu=\sigma^{2}=\theta$ yields $$ \ell(\theta)=-\frac{n}{2}\log(2\pi)-\frac{n}{2}\log(\theta)-\frac{1}{2\theta}\sum_{i=1}^{n}(x_{i}-\theta)^{2}. $$ Expanding the term under the sum leads to $$ \begin{align} \ell(\theta) &=-\frac{n}{2}\log(2\pi)-\frac{n}{2}\log(\theta)-\frac{1}{2\theta}\left(\sum_{i=1}^{n}x_{i}^{2}-2\theta\sum_{i=1}^{n}x_{i}+n\theta^{2}\right) \\ &=-\frac{n}{2}\log(2\pi)-\frac{n}{2}\log(\theta)-\frac{s}{2\theta}+t-\frac{n\theta}{2} \\ \end{align} $$ where $s=\sum_{i=1}^{n}x_{i}^{2}$ and $t=\sum_{i=1}^{n}x_{i}$. Taking the first derivative wrt $\theta$ gives $$ S(\theta)=\frac{d}{d\theta}\ell(\theta)=\frac{s}{2\theta^{2}}-\frac{n}{2\theta}-\frac{n}{2}. $$ So $s$ is the minimal sufficient statistic. The maximum likelihood estimator $\hat{\theta}$ can be found by setting $S(\theta)=0$ and solving for $\theta$.

Applying the same procedure to $\mathcal{N}(\mu=\theta, \sigma^{2}=\theta^{2})$, the log-likelihood function is $$ \ell(\theta)=-\frac{n}{2}\log(2\pi)-\frac{n}{2}\log(\theta^{2})-\frac{1}{2\theta^{2}}\sum_{i=1}^{n}(x_{i}-\theta)^{2}. $$ This leads to the following score function (again, with $s=\sum_{i=1}^{n}x_{i}^{2}$ and $t=\sum_{i=1}^{n}x_{i}$): $$ S(\theta)=\frac{s}{\theta^{3}}-\frac{t}{\theta^2}-\frac{n}{\theta}. $$

Fisher information

The Fisher information is defined as the negative second derivative of the log-likelihood function: $$ I(\theta)=-\frac{d^{2}\,\ell(\theta)}{d\,\theta^{2}}=-\frac{d\,S(\theta)}{d\,\theta}. $$ The observed Fisher information is $I(\hat{\theta})$, the Fisher information evaluated at the maximum likelihood estimate.

For the second question (d), we have: $$ I(\theta)=-\frac{d}{d\,\theta}\left(\frac{s}{2\theta^{2}}-\frac{n}{2\theta}-\frac{n}{2} \right) = \frac{s}{\theta^{3}}-\frac{n}{2\theta^{2}}. $$

And for the first question (c), we have: $$ I(\theta)=-\frac{d}{d\,\theta}\left(\frac{s}{\theta^{3}}-\frac{t}{\theta^2}-\frac{n}{\theta}\right) = \frac{3s}{\theta^{4}}-\frac{2t}{\theta^{3}}-\frac{n}{\theta^{2}}. $$ To get the observed Fisher information, plug in the maximum likelihood estimates.

Gamma distribution

It looks right to me but you don't need the sums in the expressions of the Fisher information.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.