This question has come up in another thread that I started so I thought I would get more people's opinions on it. My question is

Is the residual, e, an estimator of the error, $\epsilon$?

The reason that I ask is the following. In OLS, the variance of the residuals, $\frac{\text{RSS}}{(n - K )}$, is known as the variance of the regression (where RSS is the residual sum of squares). Similarly the square root of this variance, $\sqrt\frac{\text{RSS}}{(n - K )}$, is the standard error of the regression. Given the fact that the square root of the variance, $\frac{\text{RSS}}{(n - K )}$,is a standard error, it must mean that this variance is the variance of an estimator. We already know that it is the variance of the residuals, therefore, the residual is an estimator?? (I assume of $\epsilon$)



Certainly the residuals are some sort of estimators of $\epsilon$ (to be clear, the definition of the residual is the estimator, the observed residual is an estimate). If the model is correct, then they may sometimes be a fairly good estimate.


$e = y - \hat y = X\beta + \epsilon - X(X'X)^{-1} X'(X\beta + \epsilon) = (I - H)\epsilon$,

where $H = X(X'X)^{-1} X'$ is the hat-matrix (because it 'puts the hat on' $y$) -- also sometimes called the projection matrix.


That is, the $e$'s are each a linear combination of the $\epsilon$'s; if $1-h_{ii}$ is reasonably big relative to $\sum_{j\neq i}h_{ij}$ (if $H$ is 'small' relative to I), then most of the weight is on the $i^\textrm{th}$ error (this is frequently not the case, though).

Note that $e_i/\sqrt{1-h_{ii}}$ will have the same expectation and variance as $\epsilon_i$ and if the elements of $H$ are small, in the manner just described, will be highly correlated with it -- in fact, if I have done my algebra right, the correlation between $e_i$ and $\epsilon_i$ is actually: $\text{corr}(e_i,\epsilon_i) = \sqrt{1-h_{ii}}$.

  • $\begingroup$ Thanks for the help Glen_b. Quick question: What is the matrix H? The only matrices I know related to regression are the Projection Matrix, P, and the Residual Maker, M. And I of course. $\endgroup$ – EconStats Oct 9 '13 at 22:16
  • $\begingroup$ Definition and link added in. $\endgroup$ – Glen_b -Reinstate Monica Oct 9 '13 at 22:21
  • $\begingroup$ Cool, it's just the Projection Matrix, I've seen that before! $\endgroup$ – EconStats Oct 9 '13 at 22:29
  • $\begingroup$ For the answer above, I think there is a minor mistake that e should be (I-H)(X\beta + \epsilon) $\endgroup$ – Chen Wang May 18 '18 at 8:00
  • $\begingroup$ $(I-H)X \beta=0$ $\endgroup$ – Glen_b -Reinstate Monica May 18 '18 at 12:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.