# Why does this expression simplify as such?

I'm reading through my professor's lecture notes on the multiple linear regression model and at one point he writes the following:

$$E[(b-\beta)e']=E[(X'X)^{-1}\epsilon\epsilon'M_{[X]}].$$

In the above equation, $$b$$, $$\beta$$, $$e$$, and $$\epsilon$$ are all vectors, $$X$$ is a regressor matrix and $$M$$ is the residual maker matrix. In general, I have no idea why these expressions are equivalent, and I'm particularly confused at how the $$e$$ vector disappears and the $$\epsilon$$ vector appears.

I am assuming $$b$$ is the OLS estimate of $$\beta$$ and $$e$$ is the corresponding estimate of $$\epsilon$$. Also I believe you have a typo above in your expression, as there should be $$X'$$ in front of $$\epsilon \epsilon'$$ and behind $$(X'X)^{-1}$$.

Start with the definition of $$b$$: $$b=(X'X)^{-1}X'Y.$$ Replacing $$Y$$ with $$X\beta+\epsilon$$ in our expression above, we get $$b=(X'X)^{-1}X'(X\beta+\epsilon)=\beta+(X'X)^{-1}X'\epsilon.$$ It follows that $$b-\beta = (X'X)^{-1}X'\epsilon$$

Now turn to the defintion of $$e$$: $$e=Y-\hat{Y}=Y-Xb=Y-X(X'X)^{-1}X'Y.$$

Notice $$X(X'X)^{-1}X'$$ is the projection matrix for $$X$$, which we will denote with $$P_{[X]}$$. Replacing this in our expression for $$e,$$ we get $$e=(I-P_{[X]})Y=M_{[X]}Y.$$ Replacing $$Y$$ in the expression above with $$X\beta+\epsilon$$, we get $$e=M_{[X]}(X\beta+\epsilon)=M_{[X]}\epsilon,$$ since $$M_{[X]}X$$ is a matrix of zeros.

Post-multiplying $$b-\beta$$ with $$e'$$, we get $$(b-\beta)e'=(X'X)^{-1}X'\epsilon \epsilon' M_{[X]},$$ since $$e'=\epsilon'M_{[X]}.$$

• Ah. The key thing I was missing was what you wrote in the last line. Mar 21, 2019 at 23:36

Assuming that the coefficient estimator $$b$$ is calculated by OLS estimation, you have:

\begin{aligned} b-\beta &= (X'X)^{-1} X'y - \beta \\[6pt] &= (X'X)^{-1} X'(X \beta + \epsilon)- \beta \\[6pt] &= (X'X)^{-1} (X'X) \beta + (X'X)^{-1} X' \epsilon - \beta \\[6pt] &= \beta + (X'X)^{-1} X' \epsilon - \beta \\[6pt] &= (X'X)^{-1} X' \epsilon. \\[6pt] \end{aligned}

Presumably $$e$$ is the residual vector (different to the error vector $$\epsilon$$) so we have $$e = M_{[X]} Y = M_{[X]} \epsilon$$. Substituting this vector gives:

\begin{aligned} (b-\beta) e' &= (X'X)^{-1} X' \epsilon \ (M_{[X]} \epsilon)' \\[6pt] &= (X'X)^{-1} X' \epsilon \epsilon' M_{[X]}' \\[6pt] &= (X'X)^{-1} X' \epsilon \epsilon' M_{[X]}. \\[6pt] \end{aligned}

(The last step follows from the fact that $$M_{[X]}$$ is a symmetric matrix.) So the expression given by your professor is missing the $$X'$$ term.

• Nice, I think we both must have been typing our answers at the same time. I'm glad you also found the mistake.
– dlnB
Mar 21, 2019 at 23:11
• @dlnb: Jinx! Buy me a coke!
– Ben
Mar 21, 2019 at 23:13