2
$\begingroup$

Consider the general linear regression model: $$y_i = \beta_0 + \beta_1x_{i1} + \beta_2x_{i2} + \cdots + \beta_px_{ip} + \epsilon_i = \mathbf{x}_i^t \beta + \epsilon_i$$

where $\textbf{x}_i = (1,x_{i1},x_{i2},\cdots,x_{ip})^T$, $\beta=(\beta_0,\beta_1,\cdots,\beta_p)^T$ and $\epsilon_i$ are iid N(0,$\sigma^2$).

I would like to see a complete proof of the following identity from first principles: $$\sum_{i=1}^n(y_i - \bar{y})^2 = \sum_{i=1}^n(\hat{y}_i - \bar{y})^2 + \sum_{i=1}^n(y_i - \hat{y}_i)^2$$ where $\hat{y}_i= \mathbf{x}_i^t \hat{\beta} $ ($\hat{\beta}$ is the least square estimator, $\bar{y}$ ia the sample mean of $y_i$).

I know that the two terms on the right can be obtained by subtracting and adding $\hat{y}_i$ on the left side. But this introduces a "cross term": $$\sum_{i=1}^n2(\hat{y}_i - \bar{y})(y_i - \hat{y}_i)$$

Many texts claim that this is zero, but I have not seen a general proof of this statement. How can this be shown?

$\endgroup$
1
  • $\begingroup$ Your $\hat y$'s should presumably all have subscripts $\endgroup$
    – Glen_b
    May 3, 2014 at 17:06

2 Answers 2

1
$\begingroup$

Split like so:

$=\sum_{i=1}^n \hat{y}_i (y_i -\hat{y}_i)-\bar{y} \sum_{i=1}^n (y_i - \hat{y}_i) $

$=\sum_{i=1}^n \hat{y}_i e_i -\bar{y} \sum_{i=1}^n e_i $

(where $e_i$ is the $i$-th residual)

$=\sum_{i=1}^n \hat{y}_i e_i$

Can you do it from there?

$\endgroup$
3
  • $\begingroup$ Are $e_i$ the empirical residuals? I thought I understood why they sum to zero but I am not sure now. Can you show me explicitly? As for the remaining term, the only thing I can think of is to substitute $\hat{y}_i=\mathbf{x}_i^T \beta$, and I don't think it leads anywhere. $\endgroup$ May 4, 2014 at 9:13
  • $\begingroup$ Following the first line to the second, clearly $e_i=y_i -\hat{y}_i$. Those are linear regression residuals (and, um yes, residuals are empirical). The substitution you suggest is not an equality. $\endgroup$
    – Glen_b
    May 4, 2014 at 9:52
  • $\begingroup$ One approach: you should be able to reduce proving $\hat{y}'e=0$ to proving $X'e=0$. From there, you could just substitute something for $e$ and end up with a difference of two terms that must be equal. $\endgroup$
    – Glen_b
    May 4, 2014 at 10:07
0
$\begingroup$

Since no one else said anything...here we go.

Let $X\in \mathbb{R}_{n\times k}$, $Y\in \mathbb{R}_{n\times 1}$ and $\hat{\beta} = \min_{\beta \in \mathbb{R}_{k\times 1}} \left\| Y-X\beta \right\|^2.$

And let $e$ be the residual vector, i.e. $e = Y-X\hat{\beta} = Y-\hat{Y}$.

By the normal equations, $$X^{T}(Y-X\hat{\beta}) = X^{T}e = \vec0$$ and therefore,

$$\sum_i \hat{y}_i e_i =\hat{Y}^{T}e = \hat{\beta}^{T}X^{T}e = 0.$$

Also note that the above also implies $Y-X\hat{\beta}$ is orthogonal to $\vec{1}$ as the first column of $X$ is all ones, so $$\vec1^{T}(Y-X\hat{\beta}) = \vec1^{T}(Y-\hat{Y}) = \vec1^{T}e = \sum_i e_i =0.$$

Combining these, we have $$\sum_{i=1}^n (\hat{y}_i - \bar{y})(y_i - \hat{y}_i) = \sum_{i=1}^n \hat{y}_i(y_i-\hat{y}_i) - \sum_{i=1}^n \bar{y}(y_i-\hat{y}_i) = \sum_{i=1}^n \hat{y}_i e_i - \sum_{i=1}^n \bar{y} e_i = 0. $$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.