7
$\begingroup$

I observe $y_i=\cos(\theta)+z_i$, $i=1,\ldots,n$, where each $z_i\sim\mathcal{N}(0,\sigma^2)$ is an i.i.d. zero-mean Gaussian random variable.

I am interested in estimating $\theta\in[0,\pi]$ with minimum mean square error (MSE).

By the general scalar case of CRLB I know that

$$\tag{1}\mathbb{E}[(\theta-\hat{\theta})^2]\geq\frac{\sigma^2\sin^2\theta}{n}.$$

I am wondering if an estimator exists that comes close to the CRLB in (1). Specifically, I am looking for an estimator whose MSE goes to zero when $\theta\rightarrow0$ or $\theta\rightarrow\pi$. Any ideas?

Update: Unfortunately (1) is incorrect: the $\sin\theta$ should be moved to denominator and squared. Thus, an estimator with MSE going to zero when $\theta\rightarrow0$ or $\theta\rightarrow\pi$ does not exist. I also forgot to square the $\sin\theta$ term in the analysis of the natural estimator below. When corrected, the asymptotic MSE in (2) of this estimator matches the CRLB in (1). See the answer below.

What I tried

The "natural" estimator would just average $y_i$'s and take the inverse cosine (we must also truncate the average so that it's in $[-1,1]$ to use inverse cosine). Its MSE is:

$$\mathbb{E}[(\theta-\hat{\theta})^2]=\mathbb{E}\left[\left(\theta-\cos^{-1}\left(\cos\theta+\frac{1}{n}\sum_{i=1}^nz_i\right)\right)^2\right]$$

The noise term $\frac{1}{n}\sum_{i=1}^nz_i$ gets small as $n$ increases, so the Taylor series expansion of $\cos^{-1}(\theta+x)$ around $x=0$ yields:

$$\tag{2}\mathbb{E}[(\theta-\hat{\theta})^2]\approx \frac{\sigma^2}{n\sin\theta},$$

where the approximation is from dropping of the lower-order terms in the Taylor series. While we can also use the Taylor series expansion to show that this estimator is unbiased, MSE in (2), unlike (1), has $\sin\theta$ in the denominator, which means the error gets worse when $\theta\rightarrow0$ or $\theta\rightarrow\pi$. This seems to be an inherent problem with this estimator (and not from approximation in (2)), as evident from a numerical experiment:

enter image description here

In the figure 'numerical' plots the MSE from the numerical experiment for $n=1000$ and $\sigma^2=1$, while 'approx $\cos^{-1}$(average)' and 'CRLB' plot (2) and (1) corresponding to these values of $n$ and $\sigma^2$. I think that the truncation of the average to $[-1,1]$ is to blame, but I'm not sure how to fix this.

For reference, here is the MATLAB code used to generate the "numerical" curve above:

theta_array=linspace(0,pi,20);
for ii=1:20
theta=theta_array(ii);
z=randn(1000,1000);
average=cos(theta)+mean(n);
average(average>1)=1;
average(average<-1)=-1;
error=theta-acos(average);
mean_error(ii)=mean(error);
mse(ii)=mean(error.^2);
end
Improve this question
$\endgroup$
9
  • $\begingroup$ What about the ML estimator? $\endgroup$
    – lacerbi
    May 25, 2016 at 17:19
  • 2
    $\begingroup$ @lacerbi - the natural estimator described above is the ML estimator, unfortunately. Think of the model as $y_i = a + z_i$; the ML of $a$ is the sample mean, so the ML of $\theta$ for $\theta \in (0, \pi)$ is the inverse cosine of $\hat{a}$. $\endgroup$
    – jbowman
    May 25, 2016 at 17:41
  • $\begingroup$ @jbowman Oh, yeah, thanks. What about a Bayesian estimator then, with a uniform prior over $\theta$ and some appropriate cost function? $\endgroup$
    – lacerbi
    May 25, 2016 at 18:21
  • 3
    $\begingroup$ Unfortunately, when I derive the CRLB, I get the $\sin^2\theta$ term in the denominator of your equation 1, not in the numerator. Naturally, if my derivation is correct, it would preclude the nice property of having the MSE go to zero when $\theta \to 0$ or $\theta \to \pi$, although it wouldn't preclude achieving the CRLB. $\endgroup$
    – jbowman
    May 25, 2016 at 18:46
  • 1
    $\begingroup$ @Glen_b I moved the update that I wrote at the bottom of the question to an answer, per your request. Let me know if it should be expanded further... $\endgroup$
    – M.B.M.
    May 26, 2016 at 3:03

1 Answer 1

Reset to default
2
$\begingroup$

The CRLB in the general scalar case where we want to estimate $\theta=g(a)$, is given by:

$$\mathbb{E}(\theta-\hat{\theta})^2]\geq \frac{\left(\frac{\partial g}{\partial a}\right)^2}{I(a)}$$

where $I(a)$ is the Fisher information associated with $a$. Here $a=\cos\theta$. Since $\theta=\cos^{-1}(a)$, one must square the derivative of $\cos^{-1}(a)$ and substitute $a=\cos\theta$. One therefore gets:

$$\tag{1, corrected}\mathbb{E}[(\theta-\hat{\theta})^2]\geq\frac{\sigma^2}{n\sin^2\theta}.$$

I also forgot to square the $\sin\theta$ term when taking it out of the expectation in (2). It should read:

$$\tag{2, corrected}\mathbb{E}[(\theta-\hat{\theta})^2]\approx\frac{\sigma^2}{n\sin^2\theta}.$$

Thus, the MSE of the "natural" estimator matches CRLB asymptotically.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks -- I don't see how this would be answered otherwise -- I think there's some value in actually having an answer for this one. $\endgroup$
    – Glen_b
    May 26, 2016 at 3:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.