# What is wrong with this proof/derivation?

Regarding simple linear regression $y = a + bx + \epsilon$ where $\epsilon$ is uncorrelated, E$[\epsilon]=0$, and Var$[\epsilon]=\sigma^2$, the definition of the residual sum of squares is $SS_{Res}=\Sigma\epsilon^2$ with an expected value of E$[SS_{Res}]=(n-2)\sigma^2$.

Where am I going wrong with the following naive derivation:

E$[SS_{Res}]=$ E$[\Sigma\epsilon_i^2]$

$=\Sigma$E$[\epsilon_i^2]$        since E$[a+b]=$ E$[a]+$E$[b]$

$=\Sigma($E$[\epsilon_i])^2$        since E$[ab]=$ E$[a]$E$[b]$   if   Cov$[a,b]=0$

$=n($E$[e_i])^2$

$=0$            since E[ $\epsilon_i$ ] $= 0$ by initial assumption

• What's $Cov[\epsilon_i, \epsilon_i]$ Oct 10, 2015 at 5:31
• $\epsilon_i$ does not have covariance zero with itself Oct 10, 2015 at 5:38
• $E(\epsilon^2)$ is not equal to $E(\epsilon)^2$, and the covariance $Cov(a,a)$ is not zero.
– user83346
Oct 10, 2015 at 7:28
• The line 3 of your proof is false! Please check attentionnally <a href="stat.berkeley.edu/~aditya/resources/…> Oct 10, 2015 at 12:39
• If $a,b$ are uncorrelated, then their correlation coefficient is zero, which is equivalent to Cov$[a,b]=0$. Since $\epsilon$ is assumed to be uncorrelated, Cov$[\epsilon,\epsilon]=0$. On the otherhand, since a vector cannot be orthogonal to itself, I don't see how Cov$[a,a]$, $a \neq 0$ can ever equal zero. Thus I do not understand what Montgomery, Peck and Vining 5e mean when they use the word uncorrelated in their phrase "$\epsilon$ are uncorrelated" when setting up the simple linear regression model. Oct 10, 2015 at 16:20

The sum of squared residuals, SSRes, is NOT the sum of the squared epsilons (true errors). The epsilons are the unobserved, independent N(0,$\sigma^2$) random errors. SSRes is the sum of the squared RESIDUALS. Residuals, $e_i = y_i - b_0 - b_1*x_i$ are often used as proxies for the true errors, but they aren't equal to the true errors in the model. For instance, residuals don't have variance $\sigma^2$, in fact, they usually don't even have the same variance, and they aren't independent! So the derivation is wrong from the first line.
• Thank you for correctly pointing out that I've confused the authors' usages of $\epsilon$ and $e_i$. I need to rethink this starting from line 1, as you have pointed out. For a given sample, I can see that Var$[e_i]\neq \sigma^2$ and that Var$[e_i]$ is not constant with $x$. However I am unable to see how ${e_i}$ are not independent. That quality would seem to be inherited from ${y_i}$, which inherits it from ${\epsilon_i}$, which are assumed to be uncorrelated and iid. Perhaps I do not understand what it means for ${\epsilon_i}$ to be uncorrelated and iid. Oct 10, 2015 at 17:12
• @AlaskanRon, I can visualize the experiment you suggest, but it is not clear to me that the correct interpretation is that experiment is a demonstration that $e_i$ are correlated. Oct 10, 2015 at 22:28
• That experiment certainly shows the $e_i$ are not independent! Since that doesn't necessarily imply nonzero correlation, a (standard) calculation is needed to demonstrate the correlation really is not zero.
• Under the usual regression model, where ${\bf X}$ has 1s in the first column and $x_i$ in the 2nd column, and model ${\bf Y = X\beta + \epsilon}$ where $\epsilon$ consists of independent $N(0,\sigma^2)$ errors, the least squared estimator of $\beta$ is ${\bf (X'X)^{-1}X'Y}$. Under these assumptions, the covariance of the residuals is proportional to the identity matrix minus the hat matrix: $\sigma^2({\bf I - X(X'X)^{-1}X'})$, so the covariances can be directly read off of the off-diagonal elements of this matrix. Nov 11, 2015 at 5:44