# Variance of OLS estimator of $\theta$ in $y_n = \theta x_n + \eta_n$ compared to Cramer-Rao

From Theodoridis' Machine Learning, problem 3.7:

Derive the Cramer-Rao bound for the LS estimator, where the training data result from the model $$y_n = \theta x_n + \eta_n\text{, } \qquad n = 1, 2, \dots$$ where $x_n$ and $\eta_n$ are iid samples of a zero mean random variable, with variance $\sigma^2_x$, and a Gaussian one with zero mean and variance $\sigma^2_{\eta}$, respectively. Assume, also, that $x$ and $\eta$ are independent. Then, show that the LS estimator achieves the CR bound only asymptotically.

After a lot of work, I have that the Cramer-Rao lower bound is $$\dfrac{1}{I(\theta)} = \dfrac{(\theta^2\sigma^2_x + \sigma^2_{\eta})^2}{2N\theta^2\sigma^4_{x}}$$ where $N$ is the sample size. The OLS estimator of $\theta$ is $$\hat{\theta} = \dfrac{\sum_{n=1}^{N}x_n y_n}{\sum_{n=1}^{N}x_n^2}\text{.}$$ How does one find the variance of this, given that BOTH $x_n$ and $y_n$ have variances?

I don't like the answer at https://stats.stackexchange.com/a/105411/46427, since the formula $$\sigma^2_b = (X^{T}X)^{-1}\sigma^2_e$$ assumes that the values of $X$ are fixed and known; i.e., with no variance. Why is this so? Because since $$\hat{\boldsymbol\beta} = (X^{T}X)^{-1}X^{T}\mathbf{y}$$ we obtain $$\mathrm{Var}\left(\hat{\boldsymbol\beta}\right) = (X^{T}X)^{-1}X^{T}\mathrm{Var}\left(\mathbf{y}\right)X(X^{T}X)^{-1}=\sigma^2_e(X^{T}X)^{-1}$$ if we assume that $X$ is a constant, known matrix - which is not the case here.

## 1 Answer

To recap, we have $X \sim \mathcal N(0, \sigma^2_x I)$ and $Y|X \sim \mathcal N(\theta X, \sigma^2_\eta I)$.

First, let's confirm the expected value of $\hat \theta$: $$E(\hat \theta) = E_X\left(E_{Y|X}\left[\frac{X^TY}{X^TX} \big\vert X\right]\right) = E_X\left(\frac{X^TE_{Y|X}(Y|X)}{X^TX}\right)$$ $$= E_X\left(\theta \frac{X^TX}{X^TX}\right) = \theta.$$ This confirms that $\theta$ is still unbiased.

Now for the variance, again using the law of total expectation, we have $$E(\hat \theta^2) = E_X\left[E_{Y|X}\left(\frac{(X^TY)^2}{(X^TX)^2} \big\vert X\right)\right]$$ $$= E_X\left[\frac{1}{(X^TX)^2} X^TE_{Y|X}\left(YY^T\big\vert X\right)X\right].$$ $Var(Y|X) = \sigma^2_\eta I = E(YY^T|X) - E(Y|X)E(Y|X)^T$ so $E(YY^T|X) = \sigma^2_\eta I + \theta^2 XX^T$. This means $$E(\hat \theta^2) = E_X\left[\frac{1}{(X^TX)^2} X^T\left(\sigma^2_\eta I + \theta^2 XX^T\right)X\right]$$ $$= E_X\left[\frac{\sigma_\eta^2}{X^TX} + \theta^2\right].$$ This means $$Var(\hat \theta) = E(\hat \theta^2) - E(\hat \theta)^2 = \sigma_\eta^2 E_X\left[\frac{1}{X^TX}\right].$$

$X \sim \mathcal N(0, \sigma^2_x I) \implies \frac{1}{\sigma^2_x}X^TX \sim \chi^2_n$ so $\frac{\sigma^2_x}{X^TX}$ follows an inverse chi-squared distribution. This means $$E\left(\frac{\sigma^2_x}{X^TX}\right) = \frac{1}{n-2}$$ $$\implies Var(\hat \theta) = \frac{\sigma_\eta^2 }{\sigma^2_x} E_X\left[\frac{\sigma^2_x}{X^TX}\right] = \frac{\sigma_\eta^2}{\sigma^2_x(n-2)}.$$

Confirming this by simulation:

s2_eta <- 0.29
s2_x <- 1.55
n <- 100
tt <- 2.1  # theta

set.seed(123)
nsim <- 1000
t.hats <- numeric(nsim)
for(i in 1:nsim) {

etas <- rnorm(n, 0, sqrt(s2_eta))
xs <- rnorm(n, 0, sqrt(s2_x))
y <- tt * xs + etas

t.hats[i] <- sum(y * xs) / sum(xs * xs)

}

var(t.hats) # 0.001922613
s2_eta / ((s2_x) * (n - 2)) # 0.001909151

• I have one quick question for you... thank for you your proof, by the way. In the last part of the exercise, it states "show that the LS estimator achieves the CR bound only asymptotically." Is this just the fact that both variances tend to $0$ as the sample size $\to \infty$? – Clarinetist Dec 28 '17 at 2:18
• @Clarinetist assuming the CR bound is correct that must be the case, although that language does seem weird – jld Dec 28 '17 at 2:27
• I agree, the language is weird. But I'm not sure how else to interpret it. From checking my algebra on WolframAlpha numerous times, I'm not sure what else it would be. Oh well. Nevertheless, thanks! – Clarinetist Dec 28 '17 at 2:28
• @Clarinetist Upon further reflection $\hat \theta \to_p \theta$ which is constant so in retrospect the asymptotic variance has to be $0$. Anyway, glad this was helpful! Quick proof: by Chebyshev $P\left(|\hat \theta_n - \theta| \geq \varepsilon\right) \leq \frac{\sigma_\eta^2}{\varepsilon^2 \sigma^2_x(n-2)} \to 0$ as $n \to \infty$ – jld Dec 28 '17 at 2:30
• I think it means for finite N, it is possible that variance of the LSE can be smaller than the CR lower bound. – Statisfun Dec 28 '17 at 2:40