How to test for independence of residuals in linear model?
What can I plot? What can I look for in the plots?
Or are there some other statistics that I can compute?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ Here is a related post. $\endgroup$– lmoCommented Nov 25, 2016 at 21:00
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
6
[I thought I'd be able to find a close duplicate with a similar answer but a couple of searches didn't turn up something suitable for some reason. I'll post an answer for now but I may still locate a duplicate.]
Note that residuals are not actually independent. It's the error term that's assumed to be independent. The residuals estimate the error term but they're definitely dependent.
There are many, many ways for errors to fail to be independent, so it's quite hard to do a general test for dependence (there are a few very general dependence tests but they require very large sample sizes to pick much up; failing to find dependence with a test that has its power scattered to the four winds is not much consolation). For more typical problems, you really need to specify what kind of dependence you might be looking for. For example, if you have observations over time, you might anticipate autocorrelation, which is easy to look for via an acf plot. If you suspect some form of intra-class correlation where there's a "class" variable not in the model (or indeed any dependence due to a variable not being in the model) but you have the variable or a reasonable proxy, it's easy enough to see whether residuals relate to that variable. So if you can elucidate a likely source of dependence in your problem, that will tell you a great deal about what kinds of things to look for.
-
$\begingroup$ Not sure what you mean by "The residuals estimate the error term but they're definitely dependent." -> in an lm, the residuals are assumed to arise from an iid normal distribution, so they should be a iid normally distributed random variable, and show no other dependencies. They should definitely not be "dependent" on anything. $\endgroup$ Commented Dec 22, 2019 at 9:36
-
$\begingroup$ You're confusing the residuals with the errors. The errors are never observed and often assumed iid normal. The residuals you get to observe. They are (demonstrably) neither independent (e.g. they sum to 0) nor identically distributed (e.g. since points more outlying in x-space have more influence they pull the line toward them more, so their variance is smaller). This is addressed in numerous answers on site, but if you can't locate a question that addresses it using the search function, post a question, and you will be pointed to a correct answer. $\endgroup$– Glen_bCommented Dec 22, 2019 at 9:45
-
$\begingroup$ Alternatively, if you can ask something more specific here I may be able to clarify my answer. $\endgroup$– Glen_bCommented Dec 22, 2019 at 9:53
-
$\begingroup$ Hi Glen, thanks for clarifying, I see now what you mean, and yes, for small n the residual distribution may differ from the assumed error. I wonder if the distinction is helpful for someone that wants to do practical residual checks, but it's probably good to be precise. $\endgroup$ Commented Dec 22, 2019 at 10:13
-
$\begingroup$ Even with extremely large n the residuals may be clearly neither independent or identically distributed, unless you make specific assumptions about the model/design and/or the way the sampling is done as n increases. In my experience it's frequently important; indeed this issue (the one caused by treating residual and error as synonyms) has led to several questions here (where clearly it did matter). One application that comes up regularlly in my work, it wouldn't matter how large the sample was, the structure of the problem leads to always having to deal with this issue. $\endgroup$– Glen_bCommented Dec 22, 2019 at 10:38
$\begingroup$
$\endgroup$
When you want to check for dependence of residuals, you need something they can depend on. There are basically 2 classes of dependencies
- Residuals correlate with another variable
- Residuals correlate with other (close) residuals (autocorrelation)
For 1), it is common to plot
- Res against predicted value
- Res against predictors
You can formalize any dependency you spot with a correlation test or a regression if you want, but usually problems are visually identified.
For 2), one can use autocorrelation plots or spatial / temporal variograms. A formal analysis can be done with the usual time-series / spatial analysis methods, e.g. Durbin-Watson or a CAR model for temporal, or MORAN's I for spatial. Note the caveats of all these methods, e.g. that they usually assume homogeneity of the correlation structure, which is commonly violated.
$\begingroup$
$\endgroup$
2
When the error terms in the model are independent, the dependence of the residuals is mild. So testing that the residuals are independent or uncorrelated under the normal assumption is what is commonly done.
answered Nov 26, 2016 at 0:55
-
$\begingroup$ I really fail to see why this makes any sense... Assume normality? Outside of textbooks or simulation, I cannot think of a situation where this would be appropriate $\endgroup$– RepmatCommented Nov 29, 2016 at 13:08
-
$\begingroup$ @Repmat If one does F-testing, normality would be best. $\endgroup$– CarlCommented Jun 15, 2018 at 2:55