# Variance of OLS estimator of $\theta$ in $y_n = \theta x_n + \eta_n$ compared to Cramer-Rao

From Theodoridis' Machine Learning, problem 3.7:

Derive the Cramer-Rao bound for the LS estimator, where the training data result from the model $$y_n = \theta x_n + \eta_n\text{, } \qquad n = 1, 2, \dots$$ where $x_n$ and $\eta_n$ are iid samples of a zero mean random variable, with variance $\sigma^2_x$, and a Gaussian one with zero mean and variance $\sigma^2_{\eta}$, respectively. Assume, also, that $x$ and $\eta$ are independent. Then, show that the LS estimator achieves the CR bound only asymptotically.

After a lot of work, I have that the Cramer-Rao lower bound is $$\dfrac{1}{I(\theta)} = \dfrac{(\theta^2\sigma^2_x + \sigma^2_{\eta})^2}{2N\theta^2\sigma^4_{x}}$$ where $N$ is the sample size. The OLS estimator of $\theta$ is $$\hat{\theta} = \dfrac{\sum_{n=1}^{N}x_n y_n}{\sum_{n=1}^{N}x_n^2}\text{.}$$ How does one find the variance of this, given that BOTH $x_n$ and $y_n$ have variances?

I don't like the answer at https://stats.stackexchange.com/a/105411/46427, since the formula $$\sigma^2_b = (X^{T}X)^{-1}\sigma^2_e$$ assumes that the values of $X$ are fixed and known; i.e., with no variance. Why is this so? Because since $$\hat{\boldsymbol\beta} = (X^{T}X)^{-1}X^{T}\mathbf{y}$$ we obtain $$\mathrm{Var}\left(\hat{\boldsymbol\beta}\right) = (X^{T}X)^{-1}X^{T}\mathrm{Var}\left(\mathbf{y}\right)X(X^{T}X)^{-1}=\sigma^2_e(X^{T}X)^{-1}$$ if we assume that $X$ is a constant, known matrix - which is not the case here.

• Since it is not given that $X_i$'s are Normal, Fisher information in a single observation $(X,Y)$ can be calculated from chain rule as $I_{X,Y}(\theta)=I_{Y\mid X}(\theta)+I_X(\theta)=I_{Y\mid X}(\theta)=\frac{\sigma^2_x}{\sigma^2_\eta}$. If $n$ is the sample size, it can be argued from Slutsky's theorem that $\sqrt n(\hat\theta-\theta) \stackrel{d}\longrightarrow N\left(0,\frac{\sigma^2_\eta}{\sigma^2_x}\right)$, so that variance of $\hat\theta$ for large $n$ is $\frac{\sigma^2_\eta}{n\sigma^2_x}$= Cramer-Rao bound $\frac1{nI_{X,Y}(\theta)}$. Apr 15, 2020 at 15:25

To recap, we have $X \sim \mathcal N(0, \sigma^2_x I)$ and $Y|X \sim \mathcal N(\theta X, \sigma^2_\eta I)$.

First, let's confirm the expected value of $\hat \theta$: $$E(\hat \theta) = E_X\left(E_{Y|X}\left[\frac{X^TY}{X^TX} \big\vert X\right]\right) = E_X\left(\frac{X^TE_{Y|X}(Y|X)}{X^TX}\right)$$ $$= E_X\left(\theta \frac{X^TX}{X^TX}\right) = \theta.$$ This confirms that $\theta$ is still unbiased.

Now for the variance, again using the law of total expectation, we have $$E(\hat \theta^2) = E_X\left[E_{Y|X}\left(\frac{(X^TY)^2}{(X^TX)^2} \big\vert X\right)\right]$$ $$= E_X\left[\frac{1}{(X^TX)^2} X^TE_{Y|X}\left(YY^T\big\vert X\right)X\right].$$ $Var(Y|X) = \sigma^2_\eta I = E(YY^T|X) - E(Y|X)E(Y|X)^T$ so $E(YY^T|X) = \sigma^2_\eta I + \theta^2 XX^T$. This means $$E(\hat \theta^2) = E_X\left[\frac{1}{(X^TX)^2} X^T\left(\sigma^2_\eta I + \theta^2 XX^T\right)X\right]$$ $$= E_X\left[\frac{\sigma_\eta^2}{X^TX} + \theta^2\right].$$ This means $$Var(\hat \theta) = E(\hat \theta^2) - E(\hat \theta)^2 = \sigma_\eta^2 E_X\left[\frac{1}{X^TX}\right].$$

$X \sim \mathcal N(0, \sigma^2_x I) \implies \frac{1}{\sigma^2_x}X^TX \sim \chi^2_n$ so $\frac{\sigma^2_x}{X^TX}$ follows an inverse chi-squared distribution. This means $$E\left(\frac{\sigma^2_x}{X^TX}\right) = \frac{1}{n-2}$$ $$\implies Var(\hat \theta) = \frac{\sigma_\eta^2 }{\sigma^2_x} E_X\left[\frac{\sigma^2_x}{X^TX}\right] = \frac{\sigma_\eta^2}{\sigma^2_x(n-2)}.$$

Confirming this by simulation:

s2_eta <- 0.29
s2_x <- 1.55
n <- 100
tt <- 2.1  # theta

set.seed(123)
nsim <- 1000
t.hats <- numeric(nsim)
for(i in 1:nsim) {

etas <- rnorm(n, 0, sqrt(s2_eta))
xs <- rnorm(n, 0, sqrt(s2_x))
y <- tt * xs + etas

t.hats[i] <- sum(y * xs) / sum(xs * xs)

}

var(t.hats) # 0.001922613
s2_eta / ((s2_x) * (n - 2)) # 0.001909151

• I have one quick question for you... thank for you your proof, by the way. In the last part of the exercise, it states "show that the LS estimator achieves the CR bound only asymptotically." Is this just the fact that both variances tend to $0$ as the sample size $\to \infty$? Dec 28, 2017 at 2:18
• @Clarinetist assuming the CR bound is correct that must be the case, although that language does seem weird
– jld
Dec 28, 2017 at 2:27
• I agree, the language is weird. But I'm not sure how else to interpret it. From checking my algebra on WolframAlpha numerous times, I'm not sure what else it would be. Oh well. Nevertheless, thanks! Dec 28, 2017 at 2:28
• @Clarinetist Upon further reflection $\hat \theta \to_p \theta$ which is constant so in retrospect the asymptotic variance has to be $0$. Anyway, glad this was helpful! Quick proof: by Chebyshev $P\left(|\hat \theta_n - \theta| \geq \varepsilon\right) \leq \frac{\sigma_\eta^2}{\varepsilon^2 \sigma^2_x(n-2)} \to 0$ as $n \to \infty$
– jld
Dec 28, 2017 at 2:30
• I think it means for finite N, it is possible that variance of the LSE can be smaller than the CR lower bound. Dec 28, 2017 at 2:40