CRLB for $\theta$ using random vectors $(x,y)$

Question

Let $(x_1,y_1)(x_2,y_2)... (x_n,y_n)$ be independent pairs such that $y_i=\theta x_i+\epsilon_i$ where $x_i$ and $\epsilon_i$ are iid Normal (0,1) for $i=1,2..n$

It was previously computed that

$f(x,y)=\frac{1}{2\pi}e^{\frac{-1}{2}x^2-\frac12(y-\theta x)^2 }$

And that the MLE for $\theta$ is $\theta^*=\frac{\sum_{i=1}^nx_iy_i}{\sum_{i=1}^nx_i^2}$ which is unbiased for $\theta$

I need to know if this estimator achieves the CRLB.

Here is what I have so far:

For any estimator W(M) of $\theta$ using sample $M_i's$ of$M$ which are iid $i=1..n$, the CRLB is given by

$\frac{[\frac{d}{d\theta}E(W(M))]^2}{nE_\theta[(\frac{d}{d\theta}lnf(M|\theta)]^2}$

So the numerator is 1 since $E(W(M))=\theta$.

For the denominator, I worked it out as $n(\theta^2(2-\theta^2))$.

Is this correct? It seems iffy coz it can have negative values.

Another problem is in computing the variance of the estimator.

$Var(\theta^*)=Var(\frac{\sum_{i=1}^nx_iy_i}{\sum_{i=1}^nx_i^2}$). How do I go about this?

Answer 1

The expression of your MLE isn't quite correct. The log-likelihood takes the form,

$$ \log(L(\textbf{x},\textbf{y};\theta) = constant - \frac{1}{2}\sum\limits_{i=1}^n(y_i - \theta x_i)^2 $$

Taking the derivative and setting equal to zero gives,

$$ \frac{d}{d\theta}\log(L(\textbf{x},\textbf{y};\theta) = \sum\limits_{i=1}^n(y_i - \theta x_i)x_i = 0 \Longrightarrow \hat{\theta} = \frac{\sum\limits_{i=1}^nx_iy_i}{\sum\limits_{i=1}^nx_i^2} $$

You can verify this gives a maximum using the second derivative.

Answer 2

Ok. Here's my solution in determining if the estimator reaches the CRLB.

$Var(\theta^*)=Var(\frac{\sum_{i=1}^nx_iy_i}{\sum_{i=1}^nx_i^2})=Var(\frac{\sum_{i=1}^nx_i(\theta x_i+\epsilon_i)}{\sum_{i=1}^nx_i^2})$

$=Var(\frac{\sum_{i=1}^n(\theta x_i^2+x_i\epsilon_i)}{\sum_{i=1}^nx_i^2})$

$=Var(\frac{\theta\sum_{i=1}^n x_i^2}{\sum_{i=1}^nx_i^2}+\frac{\sum_{i=1}^nx_i\epsilon_i}{\sum_{i=1}^nx_i^2})$

$=Var(\theta+\frac{\sum_{i=1}^nx_i\epsilon_i}{\sum_{i=1}^nx_i^2})=Var(\frac{\sum_{i=1}^nx_i\epsilon_i}{\sum_{i=1}^nx_i^2})$

At this point, I can argue that since $x_i's$ and $\epsilon_i's$ are iid $\sim N(0,1)$ then $Var(\theta^*)$ is independent of $\theta$ which means that it cannot attain the CRLB which is dependent on $\theta$. That is, there will always be a theta that can make the CRLB smaller than $Var(\theta^*)$.