# Classical Normal Linear Regression model identity

The context of the following identity is in the Classical Normal Linear Regression Model, ie, $\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta}+ \boldsymbol{u}$ where $\boldsymbol{u}$ is a $n \times 1$ matrix and $u_i \sim iid.N(0, \sigma^2)$ for $i = 1, 2, \cdots, n$

Show that $(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})'(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}) = (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{b})'(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{b})+(\boldsymbol{\beta}-\boldsymbol{b})'\boldsymbol{X}'\boldsymbol{X}(\boldsymbol{\beta}-\boldsymbol{b}) \ \cdots (1)$

where:

$\boldsymbol{y}$ is a $n \times 1$ matrix

$\boldsymbol{X}$ is a $n \times k$ matrix

$\boldsymbol{\beta}$ is a $k \times 1$ matrix

$\boldsymbol{b}$ is a $k \times 1$ matrix

$rank(\boldsymbol{X}) = k$

$\boldsymbol{b} = (\boldsymbol{X}'\boldsymbol{X})^{-1}\boldsymbol{X}'\boldsymbol{y}$

Question: How do I algebraically manipulate the LHS of $(1)$ to become the RHS?

• I suggest that you try writing $y-X\beta$ as $y-Xb+Xb-X\beta$, expand and simplify, and see what you end up with. Oct 7 '14 at 15:35
• You'll also need to use orthogonality of residuals (i.e. X'e=0). The result comes out in only a few lines. Oct 7 '14 at 15:46
• @Glen_b Thank you! That did the trick. Mind posting it as an answer for acceptance?
– TeTs
Oct 7 '14 at 15:47

A pretty standard trick with these things is to try adding and subtracting a term. In this case, try writing $y−Xβ$ as $y−Xb+Xb−Xβ$, expand and simplify.

Like so:

$(y-X\beta)'(y-X\beta)=(y-Xb+Xb-X\beta)'(y-Xb+Xb-X\beta)$

$=(y-Xb)'(y-Xb+Xb-X\beta)+(b-\beta)'X'(y-Xb+Xb-X\beta)$

$=(y-Xb)'(y-Xb) +(y-Xb)'X(b-\beta)+(b-\beta)'X'(y-Xb)\\ \quad +(b-\beta)'X'X(b-\beta)$

$\hspace{2 cm}$ (And now note that $e'X = 0$ and $X'e=0$, where $e=y-Xb$)

$=(y-Xb)'(y-Xb)+0+0 +(b-\beta)'X'X(b-\beta)$

• What is "$e$" and how is it related to the "$u$" of the question? Since I provided a counterexample in a comment I would be very interested in understanding why it does not indicate you have introduced unstated assumptions in this calculation.
– whuber
Oct 7 '14 at 16:09
• @whuber The $e$ is a vector of residuals, $e=y-Xb$ Oct 7 '14 at 16:18
• @whuber I believe your counterexample contains an error. $y-X\beta$ is $u$, not $1+u$. Oct 7 '14 at 16:20
• Thanks: it contained yet another more critical error, which is I overlooked the definition of $b = 1+u$. With that in place, everything works out. To avoid confusing anyone else besides myself, I have deleted that comment. (+1)
– whuber
Oct 7 '14 at 16:31