I'm studying from C. Shalizi's lecture notes https://www.stat.cmu.edu/~cshalizi/ADAfaEPoV/ .

In the third chapter he introduces the optimal linear estimator of a random variable $Y$ conditioned to another (possibly vector) $X$: $$f(X)=\beta X,\qquad \beta = \frac{1}{\text {Cov}(X,X)}\text {Cov}(X,Y).$$

Defining the error $Y-f(X)=\epsilon$ he states that, in general, $\mathbb E(\epsilon|X)\neq 0$, which I understand.

However, at page 45 he is proving that the Ordinary Least Squares estimators $\hat \beta$ give unbiased estimates of $\beta $ (as far as I understand, without any assumption about the actual correctness of the linear model). Here's the derivation.

enter image description here

My confusion concerns the step from Eq. (2.24) to (2.25), i.e. the second $+0$. Isn't he assuming here that the conditional expectation is $\mathbb E (\epsilon \vert X)=0$? And, relatedly, why in Eq. (2.24) has the $\mathbb E(|\boldsymbol X = \boldsymbol x)$ for $\mathbb \epsilon$ been replaced by an apparently unconditional expectation mean?

After some thought I realized this is probably just an error/typo from the author, which really meant that the unconditional expectation (averaged over the data set $\boldsymbol X=\boldsymbol x$) of the $\hat \beta$ estimator is equal to $\beta$. Indeed, it doesn't make much sense to think of being able to estimate the full regression line by making repeated measurements of $Y$ for few fixed values of $X$... unless the truth is a linear model, for sure :-)

If nobody comes up with corrections or anything to add, I will add the above as answer.

  • $\begingroup$ yes. what you say is correct. formally, it seems that the author is using what is referred to as the tower property: $E(E(\epsilon | X = x)) = E(\epsilon) = 0$. $\endgroup$
    – mlofton
    Commented May 29, 2020 at 22:36

1 Answer 1


If I understand correctly what is being proven, this is just an error/typo from the author.

The linear coefficient estimator(s) $\hat \beta$ is conditionally (over the input data) unbiased only if the underlying process is truly linear.

On the other hand, $\hat \beta$ is an unconditionally unbiased estimator of the optimal linear predictor. This can be formally proved by integrating Eq. (2.24) over the marginal $X$ distribution and using the tower property as pointed out by @mlofton.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.