# Analysing the residuals themselves

As far as I know, it is possible to fit a linear regression model and then fit a second model to predict the residuals from the first model by using some other variables. By this you can understand their influence on the relation modelled with the higher level model. So the purpose is not to check the model fit, but to get a deeper understanding of the data. Unfortunately, I was not able to find any literature on this.

Is there a name for this kind of analysis? How and why is it done? What are the pros and cons? Could you provide any literature on this?

• This question is a bit outdated, but I just stumbled over it. Isn't one example (G)ARCH models and their different variations? In essence, such models assume that the squared residuals are equal to the variance of the dependent variable and and a model is fit to them. Mar 3, 2021 at 10:49

## 2 Answers

This has been referred to as residual index, although not consistently. I guess the type of analyses you subject it to would depend on your question of interest (as most result in some level of 'deeper understanding'), and so would pros and cons. Garcia-Berthou discusses cons of one example of such application as "an ad hoc sequential procedure with no statistical justification" here http://onlinelibrary.wiley.com/doi/10.1046/j.1365-2656.2001.00524.x/full In other words, if you suspect other factors are affecting the response, why not start with a model that would account for these multiple factors and their interactions. Yet, in other cases it is possible to justify, and there are valid examples of its use in spatial analyses.

• Thanks! Method seems a bit suspicious but I heard of people using it and was curious about it.
– Tim
Dec 7, 2014 at 19:58
• And I think it is difficult to do a google scholar search for it because it often is an after-thought of analyses, not prominently mentioned in keywords or abstract, so it could be more widespread than it seems. Dec 7, 2014 at 20:26
• That makes sense because I was not able to find much information on it.
– Tim
Dec 7, 2014 at 20:28
• The Garcia-Bethou paper is discussing a quite specific use for this approach, as an alternative to "analysis of covariance" for comparing treatment effects among groups with differing levels of a nuisance covariate. I don't know that his critique applies more generally. Dec 9, 2016 at 17:26

As a partial answer to your question, you can build a residual model (also known as a variance model) to model the residuals for the original model as a function of the predicted response (for example) as follows (in R notation):

residual.model <-
lm(abs(residuals(original.model)) ~ predict(original.model), ...)


You can then examine the residual model to get a deeper understanding of the data. From the residual model you can estimate prediction intervals, for example.

The residuals are usually much more noisy than the original data used to build the model, and so there will be more uncertainty about the residual model than the original model.

The above residual model uses lm and thus assumes (at most) a linear relationship between the absolute residuals and the predicted response, which is often a good enough approximation to the (unknown) underlying reality---or maybe I should say, a linear residual model is often about the most complicated residual model you would want to use given the noise in the residuals.

My earth R package builds variance models essentially using the above idea (but the ideas are fairly universal and apply not just to earth/MARS models). Some of the background theory can be found in the package vignette Variance models in earth. When reading the vignette, mentally substitute the name of your model instead of "earth" e.g. substitute "lm" for "earth".

Additional references can be found in the above vignette. Especially helpful is Carroll and Ruppert Transformation and Weighting in Regression.