From Theodoridis' Machine Learning, problem 3.7:

Derive the Cramer-Rao bound for the LS estimator, where the training data result from the model $$y_n = \theta x_n + \eta_n\text{, } \qquad n = 1, 2, \dots$$ where $x_n$ and $\eta_n$ are iid samples of a zero mean random variable, with variance $\sigma^2_x$, and a Gaussian one with zero mean and variance $\sigma^2_{\eta}$, respectively. Assume, also, that $x$ and $\eta$ are independent. Then, show that the LS estimator achieves the CR bound only asymptotically.

After a lot of work, I have that the Cramer-Rao lower bound is $$\dfrac{1}{I(\theta)} = \dfrac{(\theta^2\sigma^2_x + \sigma^2_{\eta})^2}{2N\theta^2\sigma^4_{x}}$$ where $N$ is the sample size. The OLS estimator of $\theta$ is $$\hat{\theta} = \dfrac{\sum_{n=1}^{N}x_n y_n}{\sum_{n=1}^{N}x_n^2}\text{.}$$ How does one find the variance of this, given that BOTH $x_n$ and $y_n$ have variances?

I don't like the answer at https://stats.stackexchange.com/a/105411/46427, since the formula $$\sigma^2_b = (X^{T}X)^{-1}\sigma^2_e$$ assumes that the values of $X$ are fixed and known; i.e., with no variance. Why is this so? Because since $$\hat{\boldsymbol\beta} = (X^{T}X)^{-1}X^{T}\mathbf{y}$$ we obtain $$\mathrm{Var}\left(\hat{\boldsymbol\beta}\right) = (X^{T}X)^{-1}X^{T}\mathrm{Var}\left(\mathbf{y}\right)X(X^{T}X)^{-1}=\sigma^2_e(X^{T}X)^{-1}$$ if we assume that $X$ is a constant, known matrix - which is not the case here.

  • $\begingroup$ Since it is not given that $X_i$'s are Normal, Fisher information in a single observation $(X,Y)$ can be calculated from chain rule as $I_{X,Y}(\theta)=I_{Y\mid X}(\theta)+I_X(\theta)=I_{Y\mid X}(\theta)=\frac{\sigma^2_x}{\sigma^2_\eta}$. If $n$ is the sample size, it can be argued from Slutsky's theorem that $\sqrt n(\hat\theta-\theta) \stackrel{d}\longrightarrow N\left(0,\frac{\sigma^2_\eta}{\sigma^2_x}\right)$, so that variance of $\hat\theta$ for large $n$ is $\frac{\sigma^2_\eta}{n\sigma^2_x}$= Cramer-Rao bound $\frac1{nI_{X,Y}(\theta)}$. $\endgroup$ – StubbornAtom Apr 15 '20 at 15:25

To recap, we have $X \sim \mathcal N(0, \sigma^2_x I)$ and $Y|X \sim \mathcal N(\theta X, \sigma^2_\eta I)$.

First, let's confirm the expected value of $\hat \theta$: $$ E(\hat \theta) = E_X\left(E_{Y|X}\left[\frac{X^TY}{X^TX} \big\vert X\right]\right) = E_X\left(\frac{X^TE_{Y|X}(Y|X)}{X^TX}\right) $$ $$ = E_X\left(\theta \frac{X^TX}{X^TX}\right) = \theta. $$ This confirms that $\theta$ is still unbiased.

Now for the variance, again using the law of total expectation, we have $$ E(\hat \theta^2) = E_X\left[E_{Y|X}\left(\frac{(X^TY)^2}{(X^TX)^2} \big\vert X\right)\right] $$ $$ = E_X\left[\frac{1}{(X^TX)^2} X^TE_{Y|X}\left(YY^T\big\vert X\right)X\right]. $$ $Var(Y|X) = \sigma^2_\eta I = E(YY^T|X) - E(Y|X)E(Y|X)^T$ so $E(YY^T|X) = \sigma^2_\eta I + \theta^2 XX^T$. This means $$ E(\hat \theta^2) = E_X\left[\frac{1}{(X^TX)^2} X^T\left(\sigma^2_\eta I + \theta^2 XX^T\right)X\right] $$ $$ = E_X\left[\frac{\sigma_\eta^2}{X^TX} + \theta^2\right]. $$ This means $$ Var(\hat \theta) = E(\hat \theta^2) - E(\hat \theta)^2 = \sigma_\eta^2 E_X\left[\frac{1}{X^TX}\right]. $$

$X \sim \mathcal N(0, \sigma^2_x I) \implies \frac{1}{\sigma^2_x}X^TX \sim \chi^2_n$ so $\frac{\sigma^2_x}{X^TX}$ follows an inverse chi-squared distribution. This means $$ E\left(\frac{\sigma^2_x}{X^TX}\right) = \frac{1}{n-2} $$ $$ \implies Var(\hat \theta) = \frac{\sigma_\eta^2 }{\sigma^2_x} E_X\left[\frac{\sigma^2_x}{X^TX}\right] = \frac{\sigma_\eta^2}{\sigma^2_x(n-2)}. $$

Confirming this by simulation:

s2_eta <- 0.29
s2_x <- 1.55
n <- 100
tt <- 2.1  # theta

nsim <- 1000
t.hats <- numeric(nsim)
for(i in 1:nsim) {

  etas <- rnorm(n, 0, sqrt(s2_eta))
  xs <- rnorm(n, 0, sqrt(s2_x))
  y <- tt * xs + etas

  t.hats[i] <- sum(y * xs) / sum(xs * xs)


var(t.hats) # 0.001922613
s2_eta / ((s2_x) * (n - 2)) # 0.001909151
  • $\begingroup$ I have one quick question for you... thank for you your proof, by the way. In the last part of the exercise, it states "show that the LS estimator achieves the CR bound only asymptotically." Is this just the fact that both variances tend to $0$ as the sample size $\to \infty$? $\endgroup$ – Clarinetist Dec 28 '17 at 2:18
  • $\begingroup$ @Clarinetist assuming the CR bound is correct that must be the case, although that language does seem weird $\endgroup$ – jld Dec 28 '17 at 2:27
  • $\begingroup$ I agree, the language is weird. But I'm not sure how else to interpret it. From checking my algebra on WolframAlpha numerous times, I'm not sure what else it would be. Oh well. Nevertheless, thanks! $\endgroup$ – Clarinetist Dec 28 '17 at 2:28
  • 1
    $\begingroup$ @Clarinetist Upon further reflection $\hat \theta \to_p \theta$ which is constant so in retrospect the asymptotic variance has to be $0$. Anyway, glad this was helpful! Quick proof: by Chebyshev $P\left(|\hat \theta_n - \theta| \geq \varepsilon\right) \leq \frac{\sigma_\eta^2}{\varepsilon^2 \sigma^2_x(n-2)} \to 0$ as $n \to \infty$ $\endgroup$ – jld Dec 28 '17 at 2:30
  • $\begingroup$ I think it means for finite N, it is possible that variance of the LSE can be smaller than the CR lower bound. $\endgroup$ – Statisfun Dec 28 '17 at 2:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.