2
$\begingroup$

Regarding simple linear regression $y = a + bx + \epsilon$ where $\epsilon$ is uncorrelated, E$[\epsilon]=0$, and Var$[\epsilon]=\sigma^2$, the definition of the residual sum of squares is $SS_{Res}=\Sigma\epsilon^2$ with an expected value of E$[SS_{Res}]=(n-2)\sigma^2$.

Where am I going wrong with the following naive derivation:

E$[SS_{Res}]=$ E$[\Sigma\epsilon_i^2]$

$=\Sigma$E$[\epsilon_i^2]$        since E$[a+b]=$ E$[a]+$E$[b]$

$=\Sigma($E$[\epsilon_i])^2$        since E$[ab]=$ E$[a]$E$[b]$   if   Cov$[a,b]=0$

$=n($E$[e_i])^2$

$=0$            since E[ $\epsilon_i$ ] $= 0$ by initial assumption

Improve this question
$\endgroup$
15
  • 3
    $\begingroup$ What's $Cov[\epsilon_i, \epsilon_i]$ $\endgroup$
    – jlimahaverford
    Commented Oct 10, 2015 at 5:31
  • $\begingroup$ $\epsilon_i$ does not have covariance zero with itself $\endgroup$
    – Christoph Hanck
    Commented Oct 10, 2015 at 5:38
  • $\begingroup$ $E(\epsilon^2)$ is not equal to $E(\epsilon)^2$, and the covariance $Cov(a,a)$ is not zero. $\endgroup$
    – user83346
    Commented Oct 10, 2015 at 7:28
  • $\begingroup$ The line 3 of your proof is false! Please check attentionnally <a href="stat.berkeley.edu/~aditya/resources/…> $\endgroup$
    – Yusto
    Commented Oct 10, 2015 at 12:39
  • $\begingroup$ If $a,b$ are uncorrelated, then their correlation coefficient is zero, which is equivalent to Cov$[a,b]=0$. Since $\epsilon$ is assumed to be uncorrelated, Cov$[\epsilon,\epsilon]=0$. On the otherhand, since a vector cannot be orthogonal to itself, I don't see how Cov$[a,a]$, $a \neq 0$ can ever equal zero. Thus I do not understand what Montgomery, Peck and Vining 5e mean when they use the word uncorrelated in their phrase "$\epsilon$ are uncorrelated" when setting up the simple linear regression model. $\endgroup$
    – cwackers
    Commented Oct 10, 2015 at 16:20

1 Answer 1

Reset to default
1
$\begingroup$

The sum of squared residuals, SSRes, is NOT the sum of the squared epsilons (true errors). The epsilons are the unobserved, independent N(0,$\sigma^2$) random errors. SSRes is the sum of the squared RESIDUALS. Residuals, $e_i = y_i - b_0 - b_1*x_i$ are often used as proxies for the true errors, but they aren't equal to the true errors in the model. For instance, residuals don't have variance $\sigma^2$, in fact, they usually don't even have the same variance, and they aren't independent! So the derivation is wrong from the first line.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you for correctly pointing out that I've confused the authors' usages of $\epsilon$ and $e_i$. I need to rethink this starting from line 1, as you have pointed out. For a given sample, I can see that Var$[e_i]\neq \sigma^2$ and that Var$[e_i]$ is not constant with $x$. However I am unable to see how ${e_i}$ are not independent. That quality would seem to be inherited from ${y_i}$, which inherits it from ${\epsilon_i}$, which are assumed to be uncorrelated and iid. Perhaps I do not understand what it means for ${\epsilon_i}$ to be uncorrelated and iid. $\endgroup$
    – cwackers
    Commented Oct 10, 2015 at 17:12
  • $\begingroup$ Yes, it is strange that residuals are correlated with each other. You can see this experimentally by plotting a scatterplot, then fit a line to it. If you then raise one of the data values above the line in the Y direction (increasing its residual), the least squares line will move upward also, so that all of the other residuals will change. $\endgroup$
    – AlaskaRon
    Commented Oct 10, 2015 at 21:13
  • $\begingroup$ @AlaskanRon, I can visualize the experiment you suggest, but it is not clear to me that the correct interpretation is that experiment is a demonstration that $e_i$ are correlated. $\endgroup$
    – cwackers
    Commented Oct 10, 2015 at 22:28
  • $\begingroup$ That experiment certainly shows the $e_i$ are not independent! Since that doesn't necessarily imply nonzero correlation, a (standard) calculation is needed to demonstrate the correlation really is not zero. $\endgroup$
    – whuber ♦
    Commented Nov 10, 2015 at 15:27
  • $\begingroup$ Under the usual regression model, where ${\bf X}$ has 1s in the first column and $x_i$ in the 2nd column, and model ${\bf Y = X\beta + \epsilon}$ where $\epsilon$ consists of independent $N(0,\sigma^2)$ errors, the least squared estimator of $\beta$ is ${\bf (X'X)^{-1}X'Y}$. Under these assumptions, the covariance of the residuals is proportional to the identity matrix minus the hat matrix: $\sigma^2({\bf I - X(X'X)^{-1}X'})$, so the covariances can be directly read off of the off-diagonal elements of this matrix. $\endgroup$
    – AlaskaRon
    Commented Nov 11, 2015 at 5:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.