The context of the following identity is in the Classical Normal Linear Regression Model, ie, $\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta}+ \boldsymbol{u}$ where $\boldsymbol{u}$ is a $n \times 1$ matrix and $u_i \sim iid.N(0, \sigma^2)$ for $i = 1, 2, \cdots, n$

Show that $(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})'(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}) = (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{b})'(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{b})+(\boldsymbol{\beta}-\boldsymbol{b})'\boldsymbol{X}'\boldsymbol{X}(\boldsymbol{\beta}-\boldsymbol{b}) \ \cdots (1)$


$\boldsymbol{y}$ is a $n \times 1$ matrix

$\boldsymbol{X}$ is a $n \times k$ matrix

$\boldsymbol{\beta}$ is a $k \times 1$ matrix

$\boldsymbol{b}$ is a $k \times 1$ matrix

$rank(\boldsymbol{X}) = k$

$\boldsymbol{b} = (\boldsymbol{X}'\boldsymbol{X})^{-1}\boldsymbol{X}'\boldsymbol{y}$

Question: How do I algebraically manipulate the LHS of $(1)$ to become the RHS?

  • $\begingroup$ I suggest that you try writing $y-X\beta$ as $y-Xb+Xb-X\beta$, expand and simplify, and see what you end up with. $\endgroup$
    – Glen_b
    Oct 7 '14 at 15:35
  • $\begingroup$ You'll also need to use orthogonality of residuals (i.e. X'e=0). The result comes out in only a few lines. $\endgroup$
    – Glen_b
    Oct 7 '14 at 15:46
  • $\begingroup$ @Glen_b Thank you! That did the trick. Mind posting it as an answer for acceptance? $\endgroup$
    – TeTs
    Oct 7 '14 at 15:47

A pretty standard trick with these things is to try adding and subtracting a term. In this case, try writing $y−Xβ$ as $y−Xb+Xb−Xβ$, expand and simplify.

Like so:



$=(y-Xb)'(y-Xb) +(y-Xb)'X(b-\beta)+(b-\beta)'X'(y-Xb)\\ \quad +(b-\beta)'X'X(b-\beta)$

$\hspace{2 cm}$ (And now note that $e'X = 0$ and $X'e=0$, where $e=y-Xb$)

$=(y-Xb)'(y-Xb)+0+0 +(b-\beta)'X'X(b-\beta)$

  • $\begingroup$ What is "$e$" and how is it related to the "$u$" of the question? Since I provided a counterexample in a comment I would be very interested in understanding why it does not indicate you have introduced unstated assumptions in this calculation. $\endgroup$
    – whuber
    Oct 7 '14 at 16:09
  • $\begingroup$ @whuber The $e$ is a vector of residuals, $e=y-Xb$ $\endgroup$
    – Glen_b
    Oct 7 '14 at 16:18
  • $\begingroup$ @whuber I believe your counterexample contains an error. $y-X\beta$ is $u$, not $1+u$. $\endgroup$
    – Glen_b
    Oct 7 '14 at 16:20
  • $\begingroup$ Thanks: it contained yet another more critical error, which is I overlooked the definition of $b = 1+u$. With that in place, everything works out. To avoid confusing anyone else besides myself, I have deleted that comment. (+1) $\endgroup$
    – whuber
    Oct 7 '14 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.