7
$\begingroup$

In classic OLS regression it is well-known that $(\mathbf{I}-\mathbf{H})\mathbf{y}=\mathbf{r}$, where $\mathbf{I}$ is the identity matrix, $\mathbf{H}$ is the hat matrix, $\mathbf{y}$ is the vector of response (dependent) values, and $\mathbf{r}$ is the vector of residuals.

My question is whether the scalar $\mathbf{y}^T(\mathbf{I}-\mathbf{H})\mathbf{y}=\mathbf{y}^T\mathbf{r}=\mathbf{y}^T\mathbf{y}-\mathbf{y}^T\mathbf{\hat{y}}$ has some meaningful interpretation.

$\endgroup$
13
$\begingroup$

It's the sum of squared residuals. As you say (no boldface in my notation), $(I-H)y=r$ gives the residuals, as we subtract the fitted values $Hy$ from the dependent variable $y$. Also, $I-H$ is symmetric and idempotent, i.e., $(I-H)=(I-H)'$ and $(I-H)^2=I-H$.

This follows from $H=X(X'X)^{-1}X'$, which is symmetric and idempotent itself, just like $I$.

$H$ is symmetric because of the properties $(AB)'=B'A'$ and $[A^{-1}]'=[A']^{-1}$, i.e., that when taking a transpose of the product, it's the product of the transposes in reverse order, and that the inverse of a transpose is the transpose of the inverse.

Idempotency of $H$ can be seen from $$ H^2=X(X'X)^{-1}\underbrace{X'X(X'X)^{-1}}_{=I}X'=X(X'X)^{-1}X'=H $$ Thus, $(I-H)^2=I-2H+H^2=I-2H+H=I-H$.

Hence, $$ y'(I-H)y=y'(I-H)'(I-H)y=r'r $$ As $r=(r_1,\ldots,r_n)'$, $$ r'r=(r_1,\ldots,r_n)\begin{pmatrix}r_1\\ \vdots\\r_n\end{pmatrix}=\sum_{i=1}^nr_i^2 $$ So it's the sum of the squared mistakes OLS makes. Graphically, it is the sum of the areas of the dashed squares (which look like rectangles because of the aspect ratio), where the residuals are the vertical distances between the observations and the regression line:

enter image description here

It is a useful measure of fit. For example, this quantity is a key ingredient in the $R^2$ formula.

$\endgroup$
  • $\begingroup$ Such a nice explanation... $\endgroup$ – Royi Aug 19 '15 at 20:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.