4
$\begingroup$

EDIT/ UPDATE:

I have understood CRLB & why we need it. But my problem is something else. In book Introduction to statistical Signal processing with Applications, by M.D. Srinath, P.K. Rajasekaran and R. Viswanathan,

Chap 5, Sec5.5 Properties of the estimators Eq(5.5.10):

$\textrm{Var}[\hat{\theta}_{ml} −θ]=E(\hat{\theta}_{ml} −θ)^2 \ge \frac{1}{I}$, the R.H.S in this expression represents lower bound on the variance of any unbiased estimate of $\theta$ & if satisfied with an equality then is called an efficient estimator.

So, for an estimator to be efficient, the variance of the estimation error must be less than the Cramer - Rao Lower bound (CRLB): \begin{equation} \textrm{Var}[\hat{\theta_{ml}} - \theta] \ge -\frac{1}{E\left[ \displaystyle \frac{\partial^2 \ln p(z|\theta)}{\partial \theta^2} \right]} \tag{1} \end{equation}

For the example : $Z_i = \theta + V_i$, $i =1,..,N$ where

$p(z|\theta) = \displaystyle \prod_{i=1}^{N} \frac{1}{\sqrt{2\pi \sigma^2_v}} \exp\left({-\frac{(z_i - \theta)^2}{2\sigma_v^2}}\right)$

I found out that $$\hat{\theta_{ml}} = \frac{1}{N}\sum_{i=1}^{N}z_i \tag{2}$$ and is an efficient estimator.

The value of \begin{equation}E\bigg[\frac{\partial^2 \ln p(z|\theta)}{\partial \theta^2} \bigg] = -\frac{N}{\sigma_v^2} \tag{3}\end{equation}

Question:

My problem is that in order to check if it is a minimum variance (MVU) estimator, I need to calculate the LHS of $(1)$. In book, they directly equate $(1)$ without calculating what $\textrm{Var}[\hat{\theta}_{ml} - \theta]$ actually is. How do I calculate this (LHS of $(1)$) quantity.

My confusion is should we not calculate in the first place what the variance of the estimator is in order to check if the expression of the variance is equal to 1/I ?I do not know the technique of calculating $\textrm{Var}[\hat{\theta}_{ml} −\theta]$ of the example I posted in question & hence unable to determine whether it is $1/I$ . How will I calculate separately the expression for $\textrm{Var}[\hat{\theta}_{ml} −\theta]$ . I could calculate the expression for $1/I$ as given in Eq(3). This will be helpful in cases where pdf and regularization is not straight forward as a plug in.

I tried calculating what the value of the L.H.S in $(1)$ will be but got stuck. In general

$\textrm{Var}(X) = E[X^2] - (E[X])^2$ where $X = \hat{\theta}_{ml} - \theta$.

$\textrm{Var}(\hat{\theta}_{ml}) - \textrm{Var}(\theta) = \textrm{Var}\left( \displaystyle \frac{1}{N}\sum_{i=1}^{N}z_i \right) - \textrm{Var}(\theta)$.

Then what should I do next so that I get the expression $\textrm{Var}(\hat{\theta}_{ml} - \theta) = \sigma_v^2/N$

This may sound too trivial, but in other problem exercises I am unable to calculate the variance. Can somebody please show how I can calculate the variance of the estimation error?

$\endgroup$
  • $\begingroup$ Why the $\hat{\theta}_{ml}$ ? ml for maximum likelihood ? What book are you referring to in your question ? $\endgroup$ – Gilles Nov 9 '14 at 12:38
  • $\begingroup$ @Gilles: I have updated the Question and the title as I believe it was unclear and vague. Please have a look. Thank you for your effort. $\endgroup$ – SKM Nov 9 '14 at 21:31
4
$\begingroup$

If the regularity condition is satisfied prior to $(1)$ above, that is if $E\left[\displaystyle \frac{\partial\ln p(\mathbf{z}|\theta)}{\partial\theta}\right] = 0$ for all $\theta$.

Then an unbiased estimator attaining CRLB may be found if and only if \begin{equation} \displaystyle \frac{\partial\ln p(\mathbf{z}|\theta)}{\partial\theta} = I(\theta)\left(g(\mathbf{z})-\theta\right) \tag{4} \end{equation} Then the MVU estimator is $\hat{\theta} = g(\mathbf{z})$ and the minimum variance is $\displaystyle \frac{1}{I(\theta)}$ where $I(\theta)$ is the Fisher information.

Back to your question, a way of doing it is to express the partial derivative of the $PDF$ is the form in $(4)$. Or you can do partial differentiation twice and use the negative reciprocal in $(1)$ as you have it in $(3)$. But let's stick to the first method because if the CRLB is attained then $\textrm{Var}(\hat{\theta}) = \displaystyle \frac{1}{I(\theta)}$.

Allow me the change of notation from capital to small. And I'm assuming $v$ is WGN with variance $\sigma_v^2$.

\begin{align} p(\mathbf{z}|\theta) = &\displaystyle \prod_{i=1}^N\frac{1}{\sqrt{2\pi\sigma_v^2}}\exp\left[-\frac{1}{2\sigma_v^2}\left(z_i - \theta\right)^2\right]\\ =&\frac{1}{\left(2\pi\sigma_v^2\right)^{\frac{N}{2}}}\exp\left[-\frac{1}{2\sigma_v^2}\sum_{i=1}^N\left(z_i - \theta\right)^2\right]\\ \Rightarrow \frac{\partial\ln p(\mathbf{z}|\theta)}{\partial\theta}=&\frac{1}{\sigma_v^2}\sum_{i=1}^N\left(z_i - \theta\right) = \frac{N}{\sigma_v^2}\left(\bar{z} - \theta\right) \end{align}

Comparing with $(4)$, the minimum variance is ${\sigma_v^2}/N$.

EDIT:

If you still want to calculate the variance the usual way as you put it, proceed with the equation you started above. Following your notations:

\begin{align} \textrm{Var}(\hat{\theta}_{ml}) - \textrm{Var}(\theta) = & \displaystyle \textrm{Var}\left(\frac{1}{N}\sum_{i=1}^N z_i\right) - \textrm{Var}(\theta)\\ = &\frac{1}{N^2}\sum_{i=1}^N \textrm{Var}(z_i) - \textrm{Var}(\theta)\\ = & \frac{1}{N^2}(N\sigma_v^2) - \textrm{Var}(\theta)\\ = & \frac{\sigma_v^2}{N} \end{align}

$\endgroup$
  • $\begingroup$ Thank you for your reply.It is my slow understanding that I am still unable to clear my concept.Could you please clarify these>>Q1: Is variance of the estimation error = $Var(\hat{\theta})$ as mentioned in your answer or the one in text books $Var(\hat{\theta} - \theta)$? Q2: Its still unclear whether we never actually calculate the variance. Then how to conclude what the variance is and whether it is inverse of FI? Q3: It may not be always possible to get a neat expression for the first partial derivative that can be compared with (4). In which case, how to calculate the minimum variance? $\endgroup$ – SKM Nov 7 '14 at 17:39
  • $\begingroup$ My main confusion is without calculating what the expression of the variance of estimation error comes out to be, how can we say that it is equal to the inverse of Fisher information (FI)? Won't we separately calculate the FI and the variance and then see if they are equal or not?If so, then how does one calculate the variance in this case, based on the statistical formula of variance. Could you shed some light from this perspective considering that I am totally new to this subject. Thank you. $\endgroup$ – SKM Nov 7 '14 at 17:43
  • 1
    $\begingroup$ @SKM Q1. With the expression of your PDF it is the variance of the estimator. It is $\textrm{Var}(\hat{\theta})$. Q2. If the CRLB is attained , $\textrm{MVUE} \iff \textrm{Var}(\hat{\theta}) = 1/I(\theta)$, i.e you can calculate either of them, the variance or FI and one is the reciprocal (scalar case) of the other. Q3. Again, if the CRLB is attained you should be able to obtain the form in $(4)$, or an equivalent if you're not in scalar cases, and hence your minimum variance. If it's not the case, it is not the minimum variance, i.e you don't have the MVUE. $\endgroup$ – Gilles Nov 9 '14 at 12:25
  • $\begingroup$ If no MVUE is found according CRLB, check the theory on RBLS. You might want to read more on both CRLB theorem and RBLS theorem in Kay-I 1993 for more on this. $\endgroup$ – Gilles Nov 9 '14 at 13:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.