I'm studying from C. Shalizi's lecture notes https://www.stat.cmu.edu/~cshalizi/ADAfaEPoV/ .

In the third chapter he introduces the optimal linear estimator of a random variable $Y$ conditioned to another (possibly vector) $X$: $$f(X)=\beta X,\qquad \beta = \frac{1}{\text {Cov}(X,X)}\text {Cov}(X,Y).$$

Defining the error $Y-f(X)=\epsilon$ he states that, in general, $\mathbb E(\epsilon|X)\neq 0$, which I understand.

However, at page 45 he is proving that the Ordinary Least Squares estimators $\hat \beta$ give unbiased estimates of $\beta $ (as far as I understand, without any assumption about the actual correctness of the linear model). Here's the derivation.

enter image description here

My confusion concerns the step from Eq. (2.24) to (2.25), i.e. the second $+0$. Isn't he assuming here that the conditional expectation is $\mathbb E (\epsilon \vert X)=0$? And, relatedly, why in Eq. (2.24) has the $\mathbb E(|\boldsymbol X = \boldsymbol x)$ for $\mathbb \epsilon$ been replaced by an apparently unconditional expectation mean?

After some thought I realized this is probably just an error/typo from the author, which really meant that the unconditional expectation (averaged over the data set $\boldsymbol X=\boldsymbol x$) of the $\hat \beta$ estimator is equal to $\beta$. Indeed, it doesn't make much sense to think of being able to estimate the full regression line by making repeated measurements of $Y$ for few fixed values of $X$... unless the truth is a linear model, for sure :-)

If nobody comes up with corrections or anything to add, I will add the above as answer.

  • $\begingroup$ yes. what you say is correct. formally, it seems that the author is using what is referred to as the tower property: $E(E(\epsilon | X = x)) = E(\epsilon) = 0$. $\endgroup$
    – mlofton
    May 29 '20 at 22:36

If I understand correctly what is being proven, this is just an error/typo from the author.

The linear coefficient estimator(s) $\hat \beta$ is conditionally (over the input data) unbiased only if the underlying process is truly linear.

On the other hand, $\hat \beta$ is an unconditionally unbiased estimator of the optimal linear predictor. This can be formally proved by integrating Eq. (2.24) over the marginal $X$ distribution and using the tower property as pointed out by @mlofton.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.