8
$\begingroup$

This question has come up in another thread that I started so I thought I would get more people's opinions on it. My question is

Is the residual, e, an estimator of the error, $\epsilon$?

The reason that I ask is the following. In OLS, the variance of the residuals, $\frac{\text{RSS}}{(n - K )}$, is known as the variance of the regression (where RSS is the residual sum of squares). Similarly the square root of this variance, $\sqrt\frac{\text{RSS}}{(n - K )}$, is the standard error of the regression. Given the fact that the square root of the variance, $\frac{\text{RSS}}{(n - K )}$,is a standard error, it must mean that this variance is the variance of an estimator. We already know that it is the variance of the residuals, therefore, the residual is an estimator?? (I assume of $\epsilon$)

Thoughts??

$\endgroup$
10
$\begingroup$

Certainly the residuals are some sort of estimators of $\epsilon$ (to be clear, the definition of the residual is the estimator, the observed residual is an estimate). If the model is correct, then they may sometimes be a fairly good estimate.

Indeed

$e = y - \hat y = X\beta + \epsilon - X(X'X)^{-1} X'(X\beta + \epsilon) = (I - H)\epsilon$,

where $H = X(X'X)^{-1} X'$ is the hat-matrix (because it 'puts the hat on' $y$) -- also sometimes called the projection matrix.

http://en.wikipedia.org/wiki/Hat_matrix

That is, the $e$'s are each a linear combination of the $\epsilon$'s; if $1-h_{ii}$ is reasonably big relative to $\sum_{j\neq i}h_{ij}$ (if $H$ is 'small' relative to I), then most of the weight is on the $i^\textrm{th}$ error (this is frequently not the case, though).

Note that $e_i/\sqrt{1-h_{ii}}$ will have the same expectation and variance as $\epsilon_i$ and if the elements of $H$ are small, in the manner just described, will be highly correlated with it -- in fact, if I have done my algebra right, the correlation between $e_i$ and $\epsilon_i$ is actually: $\text{corr}(e_i,\epsilon_i) = \sqrt{1-h_{ii}}$.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ For the answer above, I think there is a minor mistake that e should be (I-H)(X\beta + \epsilon) $\endgroup$ – Chen Wang May 18 '18 at 8:00
  • 1
    $\begingroup$ $(I-H)X \beta=0$ $\endgroup$ – Glen_b May 18 '18 at 12:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.