# Decomposing total sum of squares

Consider the general linear regression model: $$y_i = \beta_0 + \beta_1x_{i1} + \beta_2x_{i2} + \cdots + \beta_px_{ip} + \epsilon_i = \mathbf{x}_i^t \beta + \epsilon_i$$

where $\textbf{x}_i = (1,x_{i1},x_{i2},\cdots,x_{ip})^T$, $\beta=(\beta_0,\beta_1,\cdots,\beta_p)^T$ and $\epsilon_i$ are iid N(0,$\sigma^2$).

I would like to see a complete proof of the following identity from first principles: $$\sum_{i=1}^n(y_i - \bar{y})^2 = \sum_{i=1}^n(\hat{y}_i - \bar{y})^2 + \sum_{i=1}^n(y_i - \hat{y}_i)^2$$ where $\hat{y}_i= \mathbf{x}_i^t \hat{\beta}$ ($\hat{\beta}$ is the least square estimator, $\bar{y}$ ia the sample mean of $y_i$).

I know that the two terms on the right can be obtained by subtracting and adding $\hat{y}_i$ on the left side. But this introduces a "cross term": $$\sum_{i=1}^n2(\hat{y}_i - \bar{y})(y_i - \hat{y}_i)$$

Many texts claim that this is zero, but I have not seen a general proof of this statement. How can this be shown?

• Your $\hat y$'s should presumably all have subscripts May 3, 2014 at 17:06

Split like so:

$=\sum_{i=1}^n \hat{y}_i (y_i -\hat{y}_i)-\bar{y} \sum_{i=1}^n (y_i - \hat{y}_i)$

$=\sum_{i=1}^n \hat{y}_i e_i -\bar{y} \sum_{i=1}^n e_i$

(where $e_i$ is the $i$-th residual)

$=\sum_{i=1}^n \hat{y}_i e_i$

Can you do it from there?

• Are $e_i$ the empirical residuals? I thought I understood why they sum to zero but I am not sure now. Can you show me explicitly? As for the remaining term, the only thing I can think of is to substitute $\hat{y}_i=\mathbf{x}_i^T \beta$, and I don't think it leads anywhere. May 4, 2014 at 9:13
• Following the first line to the second, clearly $e_i=y_i -\hat{y}_i$. Those are linear regression residuals (and, um yes, residuals are empirical). The substitution you suggest is not an equality. May 4, 2014 at 9:52
• One approach: you should be able to reduce proving $\hat{y}'e=0$ to proving $X'e=0$. From there, you could just substitute something for $e$ and end up with a difference of two terms that must be equal. May 4, 2014 at 10:07

Since no one else said anything...here we go.

Let $$X\in \mathbb{R}_{n\times k}$$, $$Y\in \mathbb{R}_{n\times 1}$$ and $$\hat{\beta} = \min_{\beta \in \mathbb{R}_{k\times 1}} \left\| Y-X\beta \right\|^2.$$

And let $$e$$ be the residual vector, i.e. $$e = Y-X\hat{\beta} = Y-\hat{Y}$$.

By the normal equations, $$X^{T}(Y-X\hat{\beta}) = X^{T}e = \vec0$$ and therefore,

$$\sum_i \hat{y}_i e_i =\hat{Y}^{T}e = \hat{\beta}^{T}X^{T}e = 0.$$

Also note that the above also implies $$Y-X\hat{\beta}$$ is orthogonal to $$\vec{1}$$ as the first column of $$X$$ is all ones, so $$\vec1^{T}(Y-X\hat{\beta}) = \vec1^{T}(Y-\hat{Y}) = \vec1^{T}e = \sum_i e_i =0.$$

Combining these, we have $$\sum_{i=1}^n (\hat{y}_i - \bar{y})(y_i - \hat{y}_i) = \sum_{i=1}^n \hat{y}_i(y_i-\hat{y}_i) - \sum_{i=1}^n \bar{y}(y_i-\hat{y}_i) = \sum_{i=1}^n \hat{y}_i e_i - \sum_{i=1}^n \bar{y} e_i = 0.$$