# Cramér-Rao Lower Bound for Exponential Families

I am having a problem with applying the Cramér-Rao inequality to identify the lower bound for the variance of an unbiased estimator and hoped that you guys could help me. The problem is the following:

Let $X_{1},\dots,X_{n}$ be a random sample from the density $f(x|\theta)=\frac{1}{\theta}e^{-\frac{x}{\theta}}, x\geq0$, $0$ otherwise. Find the Cramér-Rao lower bound for the variance of an unbiased estimater of the population variance $\theta^{2}$.

Now, the problem itself isn't too difficult and finding the MLE for $\theta^{2}$ wasn't a problem either. By finding the MLE for $\theta^{2}$ (which I found to be $(\frac{\sum^{n}_{i=1}Xi}{n})^{2} = \bar{X}^{2}$ if I am not mistaken), I already found log$f(x|\theta)$ and its derivatives. So since the Cramér-Rao Inequality (for iid random variables) is given by

$Var_{\theta}[W(\boldsymbol{X})] \geq \frac{(\frac{d}{d\theta}E_{\theta}W(\boldsymbol{X}))^{2}}{nE_{\theta}[(\frac{\partial}{\partial\theta}\text{log}f(\boldsymbol{X}|\theta))^{2}]}$

Also, since $f(\boldsymbol{x|\theta})$ belongs to the exponential family,

$E_{\theta}[(\frac{\partial}{\partial\theta}\text{log}f(\boldsymbol{X|\theta}))^{2}] = -E_{\theta}[\frac{\partial^{2}}{\partial^{2}\theta^{2}}\text{log}f(\boldsymbol{X|\theta})]$

If I am not totally mistaken with the notation here (we discussed this topic in class very briefly), I found the following values:

• For $E_{\theta}[\frac{\partial^{2}}{\partial^{2}\theta^{2}}\text{log}f(\boldsymbol{X|\theta})]$:

$E_{\theta}[\frac{n}{\theta^{2}} - \frac{2\sum^{n}_{i=1}X_{i}}{\theta^{3}}] = \frac{n}{\theta^{2}} - \frac{2}{\theta^{3}}E[\sum^{n}_{i=1}X_{i}] \\ = \frac{n}{\theta^{2}} - \frac{2}{\theta^{3}}nE[X_{1}] = \frac{n}{\theta^{2}} - \frac{2}{\theta^{3}}n \theta = -\frac{n}{\theta^{2}}$

Thus, the term in the denominator becomes $\frac{n^{2}}{\theta^{2}}$

• For $(\frac{d}{d\theta}E_{\theta}W(\boldsymbol{X}))^{2}$:

This is the part where I am really confused. Since we're talking about an estimator of $\theta^{2}$, I would have taken the first derivative with respect to $\theta$ and squared it, to get $4\hat{\theta}^{2}$ in the numerator. This would give

$Var_{\theta}[\hat{\theta}^{2}] \geq \frac{4\hat{\theta}^{4}}{n^{2}}$

for my Cramér-Rao bound. It just seems odd, considering the rather simple expressions we usually get as solutions to our problems. Am I on the wrong track here?

Also sorry for asking such a rather simple question. The whole statistics class was just a half-semester course, so we treated some of the concepts rather superficially.

• 1) That $n^2$ in the denominator should be an $n$. 2) The C-R bound on variance isn't in terms of $\hat{\theta}$ but in terms of $\theta$; the estimated variance is found by plugging in $\hat{\theta}$ in the formula wherever $\theta$ is found. 3) Other than that, your formula is right! It really is pretty simple, only one squared term, no additions or exponentiations or any such... – jbowman Jan 29 '16 at 19:57
• @jbowman Thank you! Yeah I might have gotten confused with the notation, now that I see your explanation it makes sense. – Bernhard Kaindl Jan 29 '16 at 20:01

While your initial formulas and computations look correct, I am getting a slightly different result for the CRLB of unbiased estimators for $\theta^2$. It might be a little tricky to get the derivatives with respect to terms that do not appear in the likelihood so let me show you a shortcut.

The basic ingredient of the CRLB is the Fisher information of course. Assume then that we have the Fisher Information for a parameter $\theta$ and we wish to derive the Fisher Information for a function of $\theta$, say $g(\theta)$. In your notation, we wish to compute

$$I \left( g\left(\theta \right) \right) = E_{\theta}\left\{ \left[ \frac{\partial}{\partial g(\theta)} \log f\left(\mathbf{x};\theta\right) \right]^2 \right\}$$

would you agree? But notice what happens when we apply the chain rule along with the definition of the derivative of the inverse function,

\begin{align} I \left( g\left(\theta \right) \right) = E_{\theta}\left\{ \left[ \frac{\partial}{\partial g(\theta)} \log f\left(\mathbf{x};\theta\right) \right]^2 \right\} & = E_{\theta} \left\{ \left[ \frac{\partial}{\partial \theta} \log f\left(\mathbf{x};\theta\right) \frac{\partial \theta}{\partial g\left( \theta \right)} \right]^2 \right\} \\ & = E_{\theta} \left\{ \left[ \frac{\partial}{\partial \theta} \log f\left(\mathbf{x};\theta\right) \frac{1}{ g ^{\prime}\left( \theta \right)} \right]^2 \right\} \\ & = \frac{I(\theta)}{\left[g ^{\prime}\left( \theta \right) \right]^2} \end{align}

which simplifies matters. With this definition, if we are looking for the Information for $\theta^2$, then since $g^{\prime} (\theta) = 2\theta$ and $I(\theta) = \frac{1}{\theta^2}$, we see that

$$I\left(\theta^2 \right) = \frac{1}{\theta^2} \frac{1}{4\theta^2}$$

Multiply by $n$ and take the reciprocal of this to arrive at your bound, $\frac{4 \theta^4}{n}$.

• Thank you for your answer! Unfortunately, we didn't have the chance to talk about the Fisher information matrix, and while I came across a similar shortcut while looking for answers, I couldn't quite wrap my head around it! But I'll try and see if I can reproduce your results. Thanks a lot – Bernhard Kaindl Jan 29 '16 at 20:07
• @BernhardKaindl This is a scalar result, did I write matrix anywhere? You also derived a Fisher Information in your question, of course you know it! – JohnK Jan 29 '16 at 20:09
• Yeah sure that makes more sense. Seems like I had been stuck with this problem for too long now. Can I just ask you a final question? As I said, I might have gotten a bit confused with the notation too, but do I need to use the joint pdf of the whole sample or just the density function given in the problem (which would make more sense now that I think about it). Because then I think I know where the $n$ that I had "too much" in my result might have come from. – Bernhard Kaindl Jan 29 '16 at 20:16
• @BernhardKaindl If you have an iid sample, both ways will lead to the same result. I wrote the proof in a more formal way but I like to work with the single density as well and multiply by $n$ afterwards. – JohnK Jan 29 '16 at 20:19
• @BernhardKaindl You are very welcome. – JohnK Jan 29 '16 at 20:20