Cramér-Rao Lower Bound for Exponential Families

Question

I am having a problem with applying the Cramér-Rao inequality to identify the lower bound for the variance of an unbiased estimator and hoped that you guys could help me. The problem is the following:

Let $X_{1},\dots,X_{n}$ be a random sample from the density $f(x \mid \theta)=\frac{1}{\theta}e^{-\frac{x}{\theta}}, x\geq0$, $0$ otherwise. Find the Cramér-Rao lower bound for the variance of an unbiased estimater of the population variance $\theta^{2}$.

Now, the problem itself isn't too difficult and finding the MLE for $\theta^{2}$ wasn't a problem either. By finding the MLE for $\theta^{2}$ (which I found to be $\left(\dfrac{\sum^{n}_{i=1}Xi}{n}\right)^{2} = \bar{X}^{2}$ if I am not mistaken), I already found log$f(x \mid \theta)$ and its derivatives. So since the Cramér-Rao Inequality (for iid random variables) is given by

$Var_{\theta}[W(\boldsymbol{X})] \geq \frac{\left(\dfrac{d}{d\theta}E_{\theta}W(\boldsymbol{X})\right)^{2}}{nE_{\theta}\left[\left(\dfrac{\partial}{\partial\theta}\text{log}f(\boldsymbol{X} \mid \theta)\right)^{2} \right]}$

Also, since $f(\boldsymbol{x \mid \theta})$ belongs to the exponential family,

$E_{\theta}\left[\left(\dfrac{\partial}{\partial\theta}\text{log}f(\boldsymbol{X \mid \theta})\right)^{2}\right] = -E_{\theta}\left[\dfrac{\partial^{2}}{\partial^{2}\theta^{2}}\text{log}f(\boldsymbol{X \mid \theta})\right]$

If I am not totally mistaken with the notation here (we discussed this topic in class very briefly), I found the following values:

• For $E_{\theta}\left[\dfrac{\partial^{2}}{\partial^{2}\theta^{2}}\text{log}f(\boldsymbol{X \mid \theta}) \right]$:

$ E_{\theta}\left[ \dfrac{n}{\theta^{2}} - \dfrac{2\sum^{n}_{i=1}X_{i}}{\theta^{3}} \right] = \dfrac{n}{\theta^{2}} - \dfrac{2}{\theta^{3}}E\left[ \sum^{n}_{i=1}X_{i} \right] \\ = \dfrac{n}{\theta^{2}} - \dfrac{2}{\theta^{3}}nE[X_{1}] = \dfrac{n}{\theta^{2}} - \dfrac{2}{\theta^{3}}n \theta = -\dfrac{n}{\theta^{2}}$

Thus, the term in the denominator becomes $\dfrac{n^{2}}{\theta^{2}}$

• For $\left(\dfrac{d}{d\theta}E_{\theta}W(\boldsymbol{X})\right)^{2}$:

This is the part where I am really confused. Since we're talking about an estimator of $\theta^{2}$, I would have taken the first derivative with respect to $\theta$ and squared it, to get $4\hat{\theta}^{2}$ in the numerator. This would give

$Var_{\theta}\left[\hat{\theta}^{2}\right] \geq \dfrac{4\hat{\theta}^{4}}{n^{2}}$

for my Cramér-Rao bound. It just seems odd, considering the rather simple expressions we usually get as solutions to our problems. Am I on the wrong track here?

Also sorry for asking such a rather simple question. The whole statistics class was just a half-semester course, so we treated some of the concepts rather superficially.

Thanks in advance!

Answer 1

While your initial formulas and computations look correct, I am getting a slightly different result for the CRLB of unbiased estimators for $\theta^2$. It might be a little tricky to get the derivatives with respect to terms that do not appear in the likelihood so let me show you a shortcut.

The basic ingredient of the CRLB is the Fisher information of course. Assume then that we have the Fisher Information for a parameter $\theta$ and we wish to derive the Fisher Information for a function of $\theta$, say $g(\theta)$. In your notation, we wish to compute

$$ I \left( g\left(\theta \right) \right) = E_{\theta}\left\{ \left[ \frac{\partial}{\partial g(\theta)} \log f\left(\mathbf{x};\theta\right) \right]^2 \right\} $$

would you agree? But notice what happens when we apply the chain rule along with the definition of the derivative of the inverse function,

\begin{align} I \left( g\left(\theta \right) \right) = E_{\theta}\left\{ \left[ \frac{\partial}{\partial g(\theta)} \log f\left(\mathbf{x};\theta\right) \right]^2 \right\} & = E_{\theta} \left\{ \left[ \frac{\partial}{\partial \theta} \log f\left(\mathbf{x};\theta\right) \frac{\partial \theta}{\partial g\left( \theta \right)} \right]^2 \right\} \\ & = E_{\theta} \left\{ \left[ \frac{\partial}{\partial \theta} \log f\left(\mathbf{x};\theta\right) \frac{1}{ g ^{\prime}\left( \theta \right)} \right]^2 \right\} \\ & = \frac{I(\theta)}{\left[g ^{\prime}\left( \theta \right) \right]^2} \end{align}

which simplifies matters. With this definition, if we are looking for the Information for $\theta^2$, then since $g^{\prime} (\theta) = 2\theta$ and $I(\theta) = \frac{1}{\theta^2}$, we see that

$$I\left(\theta^2 \right) = \frac{1}{\theta^2} \frac{1}{4\theta^2} $$

Multiply by $n$ and take the reciprocal of this to arrive at your bound, $\frac{4 \theta^4}{n}$.

Answer 2

Using this theorem from Bruce Hansen's Book(https://www.ssc.wisc.edu/~bhansen/probability/), we can directly get the result.

Assume $\hat{\beta}$ is an unbiased estimator for variance $\theta^2$, that is $$E[\hat{\beta}]=\beta=\theta^2,$$ the corresponding function is $\beta=h(\theta)=\theta^2$.

So $H=h'(\theta)=2\theta$, then using the theorem above, we get

$var(\hat{\beta})=H'(nI_{\theta})^{-1}H=2\theta(\theta^2/n)2\theta=4\theta^4/n$.