Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I was looking at the Draper & Smith's 'Applied Regression Analysis', as did the person who asked:this other question on CrossValidated

In short - the variance-covariance matrix of the residuals in regression is given by $(I - H)\sigma^2$, where $H$ is the 'Hat Matrix'. So in general we must assume the residuals are not independent.

Yet whenever I read about the assumptions of regression it says the error terms should be independent (as well as having zero expectation $E[\epsilon_i] = 0$ and equal variance $Var(\epsilon_i) = \sigma^2 \forall i$.

I am missing something, or using a loose definition - my naive, inexperienced reading of this looks contradictory. Thank you, Chris

share|improve this question
up vote 3 down vote accepted

Both of these facts are true. The errors are independent but the residuals are not. These should not be confused, however. The errors are unobservable while the residuals are something of our own construction. We hope that they behave similarly so as to make inference, but that is not always the case.

To understand the dependence of the residuals consider a model with an intercept at which case it is possible to show that the residuals should sum to zero This means that they cannot be uncorrelated as they should always satisfy this condition and deviations from it are not allowed. This correlation will tend to be more pronounced for small samples, for instance consider the case of $n=3$. We then should have

$$e_1 + e_2 + e_3 = 0 \implies e_1 = - e_2 - e_3$$

and it goes without saying that this will result in non-trivial correlations.

But then is it really wise to use the dependent residuals as a proxy for the independent errors? Well, strictly speaking no but the correlation becomes less of a problem as the sample size increases. It never really goes away but you can essentially disregard it and treat the residuals as the errors.

share|improve this answer
    
Thank you for your help, I think I need to go away and think about this for a while! – clumb Jan 11 at 19:09

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.