5
$\begingroup$

I am having a problem with applying the Cramér-Rao inequality to identify the lower bound for the variance of an unbiased estimator and hoped that you guys could help me. The problem is the following:

Let $X_{1},\dots,X_{n}$ be a random sample from the density $f(x|\theta)=\frac{1}{\theta}e^{-\frac{x}{\theta}}, x\geq0$, $0$ otherwise. Find the Cramér-Rao lower bound for the variance of an unbiased estimater of the population variance $\theta^{2}$.

Now, the problem itself isn't too difficult and finding the MLE for $\theta^{2}$ wasn't a problem either. By finding the MLE for $\theta^{2}$ (which I found to be $(\frac{\sum^{n}_{i=1}Xi}{n})^{2} = \bar{X}^{2}$ if I am not mistaken), I already found log$f(x|\theta)$ and its derivatives. So since the Cramér-Rao Inequality (for iid random variables) is given by

$Var_{\theta}[W(\boldsymbol{X})] \geq \frac{(\frac{d}{d\theta}E_{\theta}W(\boldsymbol{X}))^{2}}{nE_{\theta}[(\frac{\partial}{\partial\theta}\text{log}f(\boldsymbol{X}|\theta))^{2}]}$

Also, since $f(\boldsymbol{x|\theta})$ belongs to the exponential family,

$E_{\theta}[(\frac{\partial}{\partial\theta}\text{log}f(\boldsymbol{X|\theta}))^{2}] = -E_{\theta}[\frac{\partial^{2}}{\partial^{2}\theta^{2}}\text{log}f(\boldsymbol{X|\theta})]$

If I am not totally mistaken with the notation here (we discussed this topic in class very briefly), I found the following values:

  • For $E_{\theta}[\frac{\partial^{2}}{\partial^{2}\theta^{2}}\text{log}f(\boldsymbol{X|\theta})]$:

$ E_{\theta}[\frac{n}{\theta^{2}} - \frac{2\sum^{n}_{i=1}X_{i}}{\theta^{3}}] = \frac{n}{\theta^{2}} - \frac{2}{\theta^{3}}E[\sum^{n}_{i=1}X_{i}] \\ = \frac{n}{\theta^{2}} - \frac{2}{\theta^{3}}nE[X_{1}] = \frac{n}{\theta^{2}} - \frac{2}{\theta^{3}}n \theta = -\frac{n}{\theta^{2}}$

Thus, the term in the denominator becomes $\frac{n^{2}}{\theta^{2}}$

  • For $(\frac{d}{d\theta}E_{\theta}W(\boldsymbol{X}))^{2}$:

This is the part where I am really confused. Since we're talking about an estimator of $\theta^{2}$, I would have taken the first derivative with respect to $\theta$ and squared it, to get $4\hat{\theta}^{2}$ in the numerator. This would give

$Var_{\theta}[\hat{\theta}^{2}] \geq \frac{4\hat{\theta}^{4}}{n^{2}}$

for my Cramér-Rao bound. It just seems odd, considering the rather simple expressions we usually get as solutions to our problems. Am I on the wrong track here?

Also sorry for asking such a rather simple question. The whole statistics class was just a half-semester course, so we treated some of the concepts rather superficially.

Thanks in advance!

$\endgroup$
  • $\begingroup$ 1) That $n^2$ in the denominator should be an $n$. 2) The C-R bound on variance isn't in terms of $\hat{\theta}$ but in terms of $\theta$; the estimated variance is found by plugging in $\hat{\theta}$ in the formula wherever $\theta$ is found. 3) Other than that, your formula is right! It really is pretty simple, only one squared term, no additions or exponentiations or any such... $\endgroup$ – jbowman Jan 29 '16 at 19:57
  • $\begingroup$ @jbowman Thank you! Yeah I might have gotten confused with the notation, now that I see your explanation it makes sense. $\endgroup$ – Bernhard Kaindl Jan 29 '16 at 20:01
4
$\begingroup$

While your initial formulas and computations look correct, I am getting a slightly different result for the CRLB of unbiased estimators for $\theta^2$. It might be a little tricky to get the derivatives with respect to terms that do not appear in the likelihood so let me show you a shortcut.

The basic ingredient of the CRLB is the Fisher information of course. Assume then that we have the Fisher Information for a parameter $\theta$ and we wish to derive the Fisher Information for a function of $\theta$, say $g(\theta)$. In your notation, we wish to compute

$$ I \left( g\left(\theta \right) \right) = E_{\theta}\left\{ \left[ \frac{\partial}{\partial g(\theta)} \log f\left(\mathbf{x};\theta\right) \right]^2 \right\} $$

would you agree? But notice what happens when we apply the chain rule along with the definition of the derivative of the inverse function,

\begin{align} I \left( g\left(\theta \right) \right) = E_{\theta}\left\{ \left[ \frac{\partial}{\partial g(\theta)} \log f\left(\mathbf{x};\theta\right) \right]^2 \right\} & = E_{\theta} \left\{ \left[ \frac{\partial}{\partial \theta} \log f\left(\mathbf{x};\theta\right) \frac{\partial \theta}{\partial g\left( \theta \right)} \right]^2 \right\} \\ & = E_{\theta} \left\{ \left[ \frac{\partial}{\partial \theta} \log f\left(\mathbf{x};\theta\right) \frac{1}{ g ^{\prime}\left( \theta \right)} \right]^2 \right\} \\ & = \frac{I(\theta)}{\left[g ^{\prime}\left( \theta \right) \right]^2} \end{align}

which simplifies matters. With this definition, if we are looking for the Information for $\theta^2$, then since $g^{\prime} (\theta) = 2\theta$ and $I(\theta) = \frac{1}{\theta^2}$, we see that

$$I\left(\theta^2 \right) = \frac{1}{\theta^2} \frac{1}{4\theta^2} $$

Multiply by $n$ and take the reciprocal of this to arrive at your bound, $\frac{4 \theta^4}{n}$.

$\endgroup$
  • $\begingroup$ Thank you for your answer! Unfortunately, we didn't have the chance to talk about the Fisher information matrix, and while I came across a similar shortcut while looking for answers, I couldn't quite wrap my head around it! But I'll try and see if I can reproduce your results. Thanks a lot $\endgroup$ – Bernhard Kaindl Jan 29 '16 at 20:07
  • $\begingroup$ @BernhardKaindl This is a scalar result, did I write matrix anywhere? You also derived a Fisher Information in your question, of course you know it! $\endgroup$ – JohnK Jan 29 '16 at 20:09
  • $\begingroup$ Yeah sure that makes more sense. Seems like I had been stuck with this problem for too long now. Can I just ask you a final question? As I said, I might have gotten a bit confused with the notation too, but do I need to use the joint pdf of the whole sample or just the density function given in the problem (which would make more sense now that I think about it). Because then I think I know where the $n$ that I had "too much" in my result might have come from. $\endgroup$ – Bernhard Kaindl Jan 29 '16 at 20:16
  • $\begingroup$ @BernhardKaindl If you have an iid sample, both ways will lead to the same result. I wrote the proof in a more formal way but I like to work with the single density as well and multiply by $n$ afterwards. $\endgroup$ – JohnK Jan 29 '16 at 20:19
  • $\begingroup$ @BernhardKaindl You are very welcome. $\endgroup$ – JohnK Jan 29 '16 at 20:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.