In chapter 2 of his Advanced Data Analysis from an Elementary Point of View textbook, Cosma Shalizi tries to find the optimal linear predictor using only a random vector $\vec X$ for the variable $Y$, without using any of the classic assumptions of the linear model. As far as I can tell, the only assumptions he makes are that "multiple observations $(\vec X_i, Y_i)$ are independent for different values of $i$, with unchanging covariances".

By using as his error the sum of squared residuals, he finds the usual optimal parameters :

$$\beta = Cov(\vec X, \vec X)^{-1} Cov(\vec X, Y)$$ $$\beta_0 = E(Y) - \beta \cdot E(\vec X) $$

For $\beta$, he finds the usual estimator:

$$\hat\beta = (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top Y $$ and shows that it is consistent.

In part (page 56), he first notes that one can write : $$Y = \vec X \cdot \beta + \epsilon$$ ($\beta_0$ seems to have disappeared here: I assume he now uses centered variables ?)

He then explains that $\epsilon$ is the noise around the linear predictor. While $E(\epsilon) = 0$ and $Cov(\epsilon, \vec X)$ by construction, it is not in general true that $E(\epsilon|\vec X) = 0$ or that $Var(\epsilon|\vec X)$ is constant, as we are not within the assumptions of the linear model.

However, he then writes the following equations : $$\begin{align} \hat\beta &= (\mathbf x^\top \mathbf x)^{-1}\mathbf x^\top Y \\ &= (\mathbf x^\top \mathbf x)^{-1}\mathbf x^\top (\mathbf x \beta+\epsilon) \\ &= \beta + (\mathbf x^\top \mathbf x)^{-1}\mathbf x^\top \epsilon \end{align}$$

And then: $$\begin{align} E\left[\hat\beta|\mathbf X = \mathbf x \right] &= \beta + (\mathbf x^\top \mathbf x)^{-1}\mathbf x^\top E[\epsilon] \\ &= \beta + 0 \\ &= \beta \end{align}$$

This is the part I don't understand: how does the conditional expectation transform into the general expected value for $\epsilon$?

This result is important, because is it then used next page when he computes the unconditioned variance of $\hat \beta$ using the law of total variance (after having added several assumptions on $\epsilon$: that the errors are uncorrelated with each other, and that they're of equal variance, i.e. that $Var(\epsilon|\mathbf X = \mathbf x) = \sigma^2\mathbf I$).

First, he finds the conditioned variance: $$\begin{align} Var\left[\hat\beta|\mathbf X = \mathbf x\right] &= Var\left[(\mathbf x^\top \mathbf x)^{-1}\mathbf x^\top \epsilon|\mathbf X = \mathbf x\right] \\ &= (\mathbf x^\top \mathbf x)^{-1}\mathbf x^\top Var(\epsilon|\mathbf X = \mathbf x) \mathbf x(\mathbf x^\top \mathbf x)^{-1} \\ &= \sigma^2(\mathbf x^\top \mathbf x)^{-1} \end{align}$$

And then: $$\begin{align} Var\left[\hat\beta\right] &= E[V(\hat \beta| X)] + Var[E(\hat \beta|X)] \\ &= E[\sigma^2(\mathbf x^\top \mathbf x)^{-1}] + Var[0] \\ &= \sigma^2 E[(\mathbf x^\top \mathbf x)^{-1}] \end{align}$$

My questions are thus:

  • What am I missing in the derivation of $E\left[\hat\beta|\mathbf X = \mathbf x \right]$ shown above?
  • What can be generally be deduced about the expectation and variances, conditioned and unconditioned on $X$, of the estimator $\hat\beta$ under these weak assumptions?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.