I'm reading through my professor's lecture notes on the multiple linear regression model and at one point he writes the following:

$$E[(b-\beta)e']=E[(X'X)^{-1}\epsilon\epsilon'M_{[X]}]. $$

In the above equation, $b$, $\beta$, $e$, and $\epsilon$ are all vectors, $X$ is a regressor matrix and $M$ is the residual maker matrix. In general, I have no idea why these expressions are equivalent, and I'm particularly confused at how the $e$ vector disappears and the $\epsilon$ vector appears.


I am assuming $b$ is the OLS estimate of $\beta$ and $e$ is the corresponding estimate of $\epsilon$. Also I believe you have a typo above in your expression, as there should be $X'$ in front of $\epsilon \epsilon'$ and behind $(X'X)^{-1}$.

Start with the definition of $b$: $$b=(X'X)^{-1}X'Y.$$ Replacing $Y$ with $X\beta+\epsilon$ in our expression above, we get $$b=(X'X)^{-1}X'(X\beta+\epsilon)=\beta+(X'X)^{-1}X'\epsilon.$$ It follows that $$b-\beta = (X'X)^{-1}X'\epsilon$$

Now turn to the defintion of $e$: $$e=Y-\hat{Y}=Y-Xb=Y-X(X'X)^{-1}X'Y.$$

Notice $X(X'X)^{-1}X'$ is the projection matrix for $X$, which we will denote with $P_{[X]}$. Replacing this in our expression for $e,$ we get $$e=(I-P_{[X]})Y=M_{[X]}Y.$$ Replacing $Y$ in the expression above with $X\beta+\epsilon$, we get $$e=M_{[X]}(X\beta+\epsilon)=M_{[X]}\epsilon,$$ since $M_{[X]}X$ is a matrix of zeros.

Post-multiplying $b-\beta$ with $e'$, we get $$(b-\beta)e'=(X'X)^{-1}X'\epsilon \epsilon' M_{[X]},$$ since $e'=\epsilon'M_{[X]}.$

  • $\begingroup$ Ah. The key thing I was missing was what you wrote in the last line. $\endgroup$ – David Mar 21 at 23:36

Assuming that the coefficient estimator $b$ is calculated by OLS estimation, you have:

$$\begin{equation} \begin{aligned} b-\beta &= (X'X)^{-1} X'y - \beta \\[6pt] &= (X'X)^{-1} X'(X \beta + \epsilon)- \beta \\[6pt] &= (X'X)^{-1} (X'X) \beta + (X'X)^{-1} X' \epsilon - \beta \\[6pt] &= \beta + (X'X)^{-1} X' \epsilon - \beta \\[6pt] &= (X'X)^{-1} X' \epsilon. \\[6pt] \end{aligned} \end{equation}$$

Presumably $e$ is the residual vector (different to the error vector $\epsilon$) so we have $e = M_{[X]} Y = M_{[X]} \epsilon$. Substituting this vector gives:

$$\begin{equation} \begin{aligned} (b-\beta) e' &= (X'X)^{-1} X' \epsilon \ (M_{[X]} \epsilon)' \\[6pt] &= (X'X)^{-1} X' \epsilon \epsilon' M_{[X]}' \\[6pt] &= (X'X)^{-1} X' \epsilon \epsilon' M_{[X]}. \\[6pt] \end{aligned} \end{equation}$$

(The last step follows from the fact that $M_{[X]}$ is a symmetric matrix.) So the expression given by your professor is missing the $X'$ term.

  • 2
    $\begingroup$ Nice, I think we both must have been typing our answers at the same time. I'm glad you also found the mistake. $\endgroup$ – dlnB Mar 21 at 23:11
  • 1
    $\begingroup$ @dlnb: Jinx! Buy me a coke! $\endgroup$ – Ben Mar 21 at 23:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.