Regarding simple linear regression $y = a + bx + \epsilon$ where $\epsilon$ is uncorrelated, E$[\epsilon]=0$, and Var$[\epsilon]=\sigma^2$, the definition of the residual sum of squares is $SS_{Res}=\Sigma\epsilon^2$ with an expected value of E$[SS_{Res}]=(n-2)\sigma^2$.

Where am I going wrong with the following naive derivation:

E$[SS_{Res}]=$ E$[\Sigma\epsilon_i^2]$

$=\Sigma$E$[\epsilon_i^2]$        since E$[a+b]=$ E$[a]+$E$[b]$

$=\Sigma($E$[\epsilon_i])^2$        since E$[ab]=$ E$[a]$E$[b]$   if   Cov$[a,b]=0$


$=0$            since E[ $\epsilon_i$ ] $= 0$ by initial assumption

  • 3
    $\begingroup$ What's $Cov[\epsilon_i, \epsilon_i]$ $\endgroup$ – jlimahaverford Oct 10 '15 at 5:31
  • $\begingroup$ $\epsilon_i$ does not have covariance zero with itself $\endgroup$ – Christoph Hanck Oct 10 '15 at 5:38
  • $\begingroup$ $E(\epsilon^2)$ is not equal to $E(\epsilon)^2$, and the covariance $Cov(a,a)$ is not zero. $\endgroup$ – user83346 Oct 10 '15 at 7:28
  • $\begingroup$ The line 3 of your proof is false! Please check attentionnally <a href="stat.berkeley.edu/~aditya/resources/…> $\endgroup$ – Yusto Oct 10 '15 at 12:39
  • $\begingroup$ If $a,b$ are uncorrelated, then their correlation coefficient is zero, which is equivalent to Cov$[a,b]=0$. Since $\epsilon$ is assumed to be uncorrelated, Cov$[\epsilon,\epsilon]=0$. On the otherhand, since a vector cannot be orthogonal to itself, I don't see how Cov$[a,a]$, $a \neq 0$ can ever equal zero. Thus I do not understand what Montgomery, Peck and Vining 5e mean when they use the word uncorrelated in their phrase "$\epsilon$ are uncorrelated" when setting up the simple linear regression model. $\endgroup$ – cwackers Oct 10 '15 at 16:20

The sum of squared residuals, SSRes, is NOT the sum of the squared epsilons (true errors). The epsilons are the unobserved, independent N(0,$\sigma^2$) random errors. SSRes is the sum of the squared RESIDUALS. Residuals, $e_i = y_i - b_0 - b_1*x_i$ are often used as proxies for the true errors, but they aren't equal to the true errors in the model. For instance, residuals don't have variance $\sigma^2$, in fact, they usually don't even have the same variance, and they aren't independent! So the derivation is wrong from the first line.

| cite | improve this answer | |
  • $\begingroup$ Thank you for correctly pointing out that I've confused the authors' usages of $\epsilon$ and $e_i$. I need to rethink this starting from line 1, as you have pointed out. For a given sample, I can see that Var$[e_i]\neq \sigma^2$ and that Var$[e_i]$ is not constant with $x$. However I am unable to see how ${e_i}$ are not independent. That quality would seem to be inherited from ${y_i}$, which inherits it from ${\epsilon_i}$, which are assumed to be uncorrelated and iid. Perhaps I do not understand what it means for ${\epsilon_i}$ to be uncorrelated and iid. $\endgroup$ – cwackers Oct 10 '15 at 17:12
  • $\begingroup$ Yes, it is strange that residuals are correlated with each other. You can see this experimentally by plotting a scatterplot, then fit a line to it. If you then raise one of the data values above the line in the Y direction (increasing its residual), the least squares line will move upward also, so that all of the other residuals will change. $\endgroup$ – AlaskaRon Oct 10 '15 at 21:13
  • $\begingroup$ @AlaskanRon, I can visualize the experiment you suggest, but it is not clear to me that the correct interpretation is that experiment is a demonstration that $e_i$ are correlated. $\endgroup$ – cwackers Oct 10 '15 at 22:28
  • $\begingroup$ That experiment certainly shows the $e_i$ are not independent! Since that doesn't necessarily imply nonzero correlation, a (standard) calculation is needed to demonstrate the correlation really is not zero. $\endgroup$ – whuber Nov 10 '15 at 15:27
  • $\begingroup$ Under the usual regression model, where ${\bf X}$ has 1s in the first column and $x_i$ in the 2nd column, and model ${\bf Y = X\beta + \epsilon}$ where $\epsilon$ consists of independent $N(0,\sigma^2)$ errors, the least squared estimator of $\beta$ is ${\bf (X'X)^{-1}X'Y}$. Under these assumptions, the covariance of the residuals is proportional to the identity matrix minus the hat matrix: $\sigma^2({\bf I - X(X'X)^{-1}X'})$, so the covariances can be directly read off of the off-diagonal elements of this matrix. $\endgroup$ – AlaskaRon Nov 11 '15 at 5:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.