1
$\begingroup$

In chapter 2 of his Advanced Data Analysis from an Elementary Point of View textbook, Cosma Shalizi tries to find the optimal linear predictor using only a random vector $\vec X$ for the variable $Y$, without using any of the classic assumptions of the linear model. As far as I can tell, the only assumptions he makes are that "multiple observations $(\vec X_i, Y_i)$ are independent for different values of $i$, with unchanging covariances".

By using as his error the sum of squared residuals, he finds the usual optimal parameters :

$$\beta = Cov(\vec X, \vec X)^{-1} Cov(\vec X, Y)$$ $$\beta_0 = E(Y) - \beta \cdot E(\vec X) $$

For $\beta$, he finds the usual estimator:

$$\hat\beta = (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top Y $$ and shows that it is consistent.

In part 2.1.3.1 (page 56), he first notes that one can write : $$Y = \vec X \cdot \beta + \epsilon$$ ($\beta_0$ seems to have disappeared here: I assume he now uses centered variables ?)

He then explains that $\epsilon$ is the noise around the linear predictor. While $E(\epsilon) = 0$ and $Cov(\epsilon, \vec X)$ by construction, it is not in general true that $E(\epsilon|\vec X) = 0$ or that $Var(\epsilon|\vec X)$ is constant, as we are not within the assumptions of the linear model.

However, he then writes the following equations : $$\begin{align} \hat\beta &= (\mathbf x^\top \mathbf x)^{-1}\mathbf x^\top Y \\ &= (\mathbf x^\top \mathbf x)^{-1}\mathbf x^\top (\mathbf x \beta+\epsilon) \\ &= \beta + (\mathbf x^\top \mathbf x)^{-1}\mathbf x^\top \epsilon \end{align}$$

And then: $$\begin{align} E\left[\hat\beta|\mathbf X = \mathbf x \right] &= \beta + (\mathbf x^\top \mathbf x)^{-1}\mathbf x^\top E[\epsilon] \\ &= \beta + 0 \\ &= \beta \end{align}$$

This is the part I don't understand: how does the conditional expectation transform into the general expected value for $\epsilon$?

This result is important, because is it then used next page when he computes the unconditioned variance of $\hat \beta$ using the law of total variance (after having added several assumptions on $\epsilon$: that the errors are uncorrelated with each other, and that they're of equal variance, i.e. that $Var(\epsilon|\mathbf X = \mathbf x) = \sigma^2\mathbf I$).

First, he finds the conditioned variance: $$\begin{align} Var\left[\hat\beta|\mathbf X = \mathbf x\right] &= Var\left[(\mathbf x^\top \mathbf x)^{-1}\mathbf x^\top \epsilon|\mathbf X = \mathbf x\right] \\ &= (\mathbf x^\top \mathbf x)^{-1}\mathbf x^\top Var(\epsilon|\mathbf X = \mathbf x) \mathbf x(\mathbf x^\top \mathbf x)^{-1} \\ &= \sigma^2(\mathbf x^\top \mathbf x)^{-1} \end{align}$$

And then: $$\begin{align} Var\left[\hat\beta\right] &= E[V(\hat \beta| X)] + Var[E(\hat \beta|X)] \\ &= E[\sigma^2(\mathbf x^\top \mathbf x)^{-1}] + Var[0] \\ &= \sigma^2 E[(\mathbf x^\top \mathbf x)^{-1}] \end{align}$$

My questions are thus:

  • What am I missing in the derivation of $E\left[\hat\beta|\mathbf X = \mathbf x \right]$ shown above?
  • What can be generally be deduced about the expectation and variances, conditioned and unconditioned on $X$, of the estimator $\hat\beta$ under these weak assumptions?
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.