Variance of OLS estimator of $\theta$ in $y_n = \theta x_n + \eta_n$ compared to Cramer-Rao

Question

From Theodoridis' Machine Learning, problem 3.7:

Derive the Cramer-Rao bound for the LS estimator, where the training data result from the model $$y_n = \theta x_n + \eta_n\text{, } \qquad n = 1, 2, \dots$$ where $x_n$ and $\eta_n$ are iid samples of a zero mean random variable, with variance $\sigma^2_x$, and a Gaussian one with zero mean and variance $\sigma^2_{\eta}$, respectively. Assume, also, that $x$ and $\eta$ are independent. Then, show that the LS estimator achieves the CR bound only asymptotically.

After a lot of work, I have that the Cramer-Rao lower bound is $$\dfrac{1}{I(\theta)} = \dfrac{(\theta^2\sigma^2_x + \sigma^2_{\eta})^2}{2N\theta^2\sigma^4_{x}}$$ where $N$ is the sample size. The OLS estimator of $\theta$ is $$\hat{\theta} = \dfrac{\sum_{n=1}^{N}x_n y_n}{\sum_{n=1}^{N}x_n^2}\text{.}$$ How does one find the variance of this, given that BOTH $x_n$ and $y_n$ have variances?

I don't like the answer at https://stats.stackexchange.com/a/105411/46427, since the formula $$\sigma^2_b = (X^{T}X)^{-1}\sigma^2_e$$ assumes that the values of $X$ are fixed and known; i.e., with no variance. Why is this so? Because since $$\hat{\boldsymbol\beta} = (X^{T}X)^{-1}X^{T}\mathbf{y}$$ we obtain $$\mathrm{Var}\left(\hat{\boldsymbol\beta}\right) = (X^{T}X)^{-1}X^{T}\mathrm{Var}\left(\mathbf{y}\right)X(X^{T}X)^{-1}=\sigma^2_e(X^{T}X)^{-1}$$ if we assume that $X$ is a constant, known matrix - which is not the case here.

Answer 1

To recap, we have $X \sim \mathcal N(0, \sigma^2_x I)$ and $Y|X \sim \mathcal N(\theta X, \sigma^2_\eta I)$.

First, let's confirm the expected value of $\hat \theta$: $$ E(\hat \theta) = E_X\left(E_{Y|X}\left[\frac{X^TY}{X^TX} \big\vert X\right]\right) = E_X\left(\frac{X^TE_{Y|X}(Y|X)}{X^TX}\right) $$ $$ = E_X\left(\theta \frac{X^TX}{X^TX}\right) = \theta. $$ This confirms that $\theta$ is still unbiased.

Now for the variance, again using the law of total expectation, we have $$ E(\hat \theta^2) = E_X\left[E_{Y|X}\left(\frac{(X^TY)^2}{(X^TX)^2} \big\vert X\right)\right] $$ $$ = E_X\left[\frac{1}{(X^TX)^2} X^TE_{Y|X}\left(YY^T\big\vert X\right)X\right]. $$ $Var(Y|X) = \sigma^2_\eta I = E(YY^T|X) - E(Y|X)E(Y|X)^T$ so $E(YY^T|X) = \sigma^2_\eta I + \theta^2 XX^T$. This means $$ E(\hat \theta^2) = E_X\left[\frac{1}{(X^TX)^2} X^T\left(\sigma^2_\eta I + \theta^2 XX^T\right)X\right] $$ $$ = E_X\left[\frac{\sigma_\eta^2}{X^TX} + \theta^2\right]. $$ This means $$ Var(\hat \theta) = E(\hat \theta^2) - E(\hat \theta)^2 = \sigma_\eta^2 E_X\left[\frac{1}{X^TX}\right]. $$

$X \sim \mathcal N(0, \sigma^2_x I) \implies \frac{1}{\sigma^2_x}X^TX \sim \chi^2_n$ so $\frac{\sigma^2_x}{X^TX}$ follows an inverse chi-squared distribution. This means $$ E\left(\frac{\sigma^2_x}{X^TX}\right) = \frac{1}{n-2} $$ $$ \implies Var(\hat \theta) = \frac{\sigma_\eta^2 }{\sigma^2_x} E_X\left[\frac{\sigma^2_x}{X^TX}\right] = \frac{\sigma_\eta^2}{\sigma^2_x(n-2)}. $$

Confirming this by simulation:

s2_eta <- 0.29
s2_x <- 1.55
n <- 100
tt <- 2.1  # theta

set.seed(123)
nsim <- 1000
t.hats <- numeric(nsim)
for(i in 1:nsim) {

  etas <- rnorm(n, 0, sqrt(s2_eta))
  xs <- rnorm(n, 0, sqrt(s2_x))
  y <- tt * xs + etas

  t.hats[i] <- sum(y * xs) / sum(xs * xs)

}

var(t.hats) # 0.001922613
s2_eta / ((s2_x) * (n - 2)) # 0.001909151