3
$\begingroup$

I was looking at the Draper & Smith's 'Applied Regression Analysis', as did the person who asked:this other question on CrossValidated

In short - the variance-covariance matrix of the residuals in regression is given by $(I - H)\sigma^2$, where $H$ is the 'Hat Matrix'. So in general we must assume the residuals are not independent.

Yet whenever I read about the assumptions of regression it says the error terms should be independent (as well as having zero expectation $E[\epsilon_i] = 0$ and equal variance $Var(\epsilon_i) = \sigma^2 \forall i$.

I am missing something, or using a loose definition - my naive, inexperienced reading of this looks contradictory. Thank you, Chris

$\endgroup$
4
$\begingroup$

Both of these facts are true. The errors are independent but the residuals are not. These should not be confused, however. The errors are unobservable while the residuals are something of our own construction. We hope that they behave similarly so as to make inference, but that is not always the case.

To understand the dependence of the residuals consider a model with an intercept at which case it is possible to show that the residuals should sum to zero This means that they cannot be uncorrelated as they should always satisfy this condition and deviations from it are not allowed. This correlation will tend to be more pronounced for small samples, for instance consider the case of $n=3$. We then should have

$$e_1 + e_2 + e_3 = 0 \implies e_1 = - e_2 - e_3$$

and it goes without saying that this will result in non-trivial correlations.

But then is it really wise to use the dependent residuals as a proxy for the independent errors? Well, strictly speaking no but the correlation becomes less of a problem as the sample size increases. It never really goes away but you can essentially disregard it and treat the residuals as the errors.

$\endgroup$
  • $\begingroup$ Thank you for your help, I think I need to go away and think about this for a while! $\endgroup$ – clumb Jan 11 '16 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.