Decomposing total sum of squares

Question

Consider the general linear regression model: $$y_i = \beta_0 + \beta_1x_{i1} + \beta_2x_{i2} + \cdots + \beta_px_{ip} + \epsilon_i = \mathbf{x}_i^t \beta + \epsilon_i$$

where $\textbf{x}_i = (1,x_{i1},x_{i2},\cdots,x_{ip})^T$, $\beta=(\beta_0,\beta_1,\cdots,\beta_p)^T$ and $\epsilon_i$ are iid N(0,$\sigma^2$).

I would like to see a complete proof of the following identity from first principles: $$\sum_{i=1}^n(y_i - \bar{y})^2 = \sum_{i=1}^n(\hat{y}_i - \bar{y})^2 + \sum_{i=1}^n(y_i - \hat{y}_i)^2$$ where $\hat{y}_i= \mathbf{x}_i^t \hat{\beta} $ ($\hat{\beta}$ is the least square estimator, $\bar{y}$ ia the sample mean of $y_i$).

I know that the two terms on the right can be obtained by subtracting and adding $\hat{y}_i$ on the left side. But this introduces a "cross term": $$\sum_{i=1}^n2(\hat{y}_i - \bar{y})(y_i - \hat{y}_i)$$

Many texts claim that this is zero, but I have not seen a general proof of this statement. How can this be shown?

Answer 1

Split like so:

$=\sum_{i=1}^n \hat{y}_i (y_i -\hat{y}_i)-\bar{y} \sum_{i=1}^n (y_i - \hat{y}_i) $

$=\sum_{i=1}^n \hat{y}_i e_i -\bar{y} \sum_{i=1}^n e_i $

(where $e_i$ is the $i$-th residual)

$=\sum_{i=1}^n \hat{y}_i e_i$

Can you do it from there?

Answer 2

Since no one else said anything...here we go.

Let $X\in \mathbb{R}_{n\times k}$, $Y\in \mathbb{R}_{n\times 1}$ and $\hat{\beta} = \min_{\beta \in \mathbb{R}_{k\times 1}} \left\| Y-X\beta \right\|^2.$

And let $e$ be the residual vector, i.e. $e = Y-X\hat{\beta} = Y-\hat{Y}$.

By the normal equations, $$X^{T}(Y-X\hat{\beta}) = X^{T}e = \vec0$$ and therefore,

$$\sum_i \hat{y}_i e_i =\hat{Y}^{T}e = \hat{\beta}^{T}X^{T}e = 0.$$

Also note that the above also implies $Y-X\hat{\beta}$ is orthogonal to $\vec{1}$ as the first column of $X$ is all ones, so $$\vec1^{T}(Y-X\hat{\beta}) = \vec1^{T}(Y-\hat{Y}) = \vec1^{T}e = \sum_i e_i =0.$$

Combining these, we have $$\sum_{i=1}^n (\hat{y}_i - \bar{y})(y_i - \hat{y}_i) = \sum_{i=1}^n \hat{y}_i(y_i-\hat{y}_i) - \sum_{i=1}^n \bar{y}(y_i-\hat{y}_i) = \sum_{i=1}^n \hat{y}_i e_i - \sum_{i=1}^n \bar{y} e_i = 0. $$