Classical Normal Linear Regression model identity

Question

The context of the following identity is in the Classical Normal Linear Regression Model, ie, $\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta}+ \boldsymbol{u}$ where $\boldsymbol{u}$ is a $n \times 1$ matrix and $u_i \sim iid.N(0, \sigma^2)$ for $i = 1, 2, \cdots, n$

Show that $(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})'(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}) = (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{b})'(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{b})+(\boldsymbol{\beta}-\boldsymbol{b})'\boldsymbol{X}'\boldsymbol{X}(\boldsymbol{\beta}-\boldsymbol{b}) \ \cdots (1)$

where:

$\boldsymbol{y}$ is a $n \times 1$ matrix

$\boldsymbol{X}$ is a $n \times k$ matrix

$\boldsymbol{\beta}$ is a $k \times 1$ matrix

$\boldsymbol{b}$ is a $k \times 1$ matrix

$rank(\boldsymbol{X}) = k$

$\boldsymbol{b} = (\boldsymbol{X}'\boldsymbol{X})^{-1}\boldsymbol{X}'\boldsymbol{y}$

---

Question: How do I algebraically manipulate the LHS of $(1)$ to become the RHS?

Answer 1

A pretty standard trick with these things is to try adding and subtracting a term. In this case, try writing $y−Xβ$ as $y−Xb+Xb−Xβ$, expand and simplify.

Like so:

$(y-X\beta)'(y-X\beta)=(y-Xb+Xb-X\beta)'(y-Xb+Xb-X\beta)$

$=(y-Xb)'(y-Xb+Xb-X\beta)+(b-\beta)'X'(y-Xb+Xb-X\beta)$

$=(y-Xb)'(y-Xb) +(y-Xb)'X(b-\beta)+(b-\beta)'X'(y-Xb)\\ \quad +(b-\beta)'X'X(b-\beta)$

$\hspace{2 cm}$ (And now note that $e'X = 0$ and $X'e=0$, where $e=y-Xb$)

$=(y-Xb)'(y-Xb)+0+0 +(b-\beta)'X'X(b-\beta)$