When a parameterized data model and corresponding pdf are known, the Cramér-Rao lower bound provides an lower bound for the variance of an estimator of one of the parameters. That is, given the data model $$ x[n] = s[n;\theta] + \epsilon[n] $$ with the data model $s[n;\theta]$ and Gaussian measurement noise $\epsilon[n]$, the Cramér-Rao lower bound bounds the variance of the estimator $\hat\theta$: $$ \text{var}(\theta) \geq \dfrac{1}{\mathcal{I}(\theta)} $$ Where $\mathcal{I}(\theta)$ is the Fisher information of the parameter $\theta$. The Fisher information can be computed as $$ \mathcal{I}(\theta) = - \mathbf{E}_x \left[ \dfrac{\partial^2 \ln p(X; \theta)}{\partial\theta^2} \right] $$ For the data model specified earlier, working out the equations leads to
$$ \mathcal{I}(\theta) = \dfrac{1}{\sigma^2}\left( \dfrac{\partial s[n;\theta]}{\partial\theta} \right)^2 $$ Now lew $s[n;\theta] = \cos(\theta)$. Considering a single-sample observation, teh Cramér-Rao lower bound is found to be $$ \text{var}(\hat\theta) \geq \dfrac{\sigma^2}{\sin(\theta)^2} $$


The Fisher information suggest that for the model described above, the variance of the estimator is maximized at the zero crossings of $cos(\theta)$, and the lower bound of the estimator approaches infinity at the peaks of $s[n;\theta]$, for $\theta=k\pi/2$.

Intuitively, I would say that it is easier to estimate $\sin[\theta]$ at the maximum or minimum, since those are the only points that occur only once per period. The zero crossings have more ambiguity: given an observed zero crossing, you would not know which zero-crossing was observed.

Why does the Fisher information go to zero for some values of $\theta$? Somehow, the Fisher information does not some to include the fact that $\cos(\theta)$ is in itself not a one-to-one function, how can I include this in estimating the parameters?

  • $\begingroup$ The short answers to your questions are 1) calculus. That's the way the math works. 2) Just restrict the range of the estimate of $\theta$ to $[-\pi, \pi)$ in the optimization routine. Note that Fisher information is calculated at the true $\theta$, so the fact that $\cos \theta$ is not 1-1 is irrelevant; there's only one value, that of $\theta$, that is relevant, and that's the true value. $\endgroup$ – jbowman Jun 14 '18 at 15:09
  • $\begingroup$ Thanks for your reply! I can see that it follows from the equations as d/dx $cos(x)$ is zero in this part, but am looking for a more intuitive understanding. Why does a zero derivative imply no Fisher information? It appears to me that the Fisher information only accounts for local variance of the estimate, but not the global confusion, e.g. when $cos(\theta) = 0$. So for a practical estimator, the variance will not match the CRLB in that case, I would say $\endgroup$ – RvdV Jun 15 '18 at 8:02
  • $\begingroup$ I think you can think of it this way. An infinitesimal change in $\theta$ doesn't change $\cos(\theta)$, therefore doesn't change the value of $f(x)$ for any $x$. Consequently, $x$ conveys no information about $\theta$ (infinitesimally), so the Fisher information is 0. OTOH, at the point where $\cos(\theta)$ crosses the axis, a change in $\theta$ causes the maximum possible change in $\cos(\theta)$, therefore, in an infinitesimal sense, the maximum possible change in $f(x)$, so Fisher information is maximized. That seems right to me, but others may have a different take on it. $\endgroup$ – jbowman Jun 15 '18 at 13:28
  • $\begingroup$ Thanks again for your response, that was my conclusion thus far too. I guess I need to address the problem of the function not being one-to-one separately, since FI only addresses the 'local' variance of the estimator / sensitivity of s[n]. $\endgroup$ – RvdV Jun 19 '18 at 12:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.