Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

According to Regression Analysis by Example, the residual is the difference between response and predicted value, then it is said that every residual has different variance, so we need to consider standardized residuals.

But the variance is for a group of values, how could a single value have variance?

share|improve this question
It would help to quote the textbook directly or (if it is available online) to provide a link to it. Much can get lost if even a single word is taken out of order or out of context. (For instance, residuals are usually defined as the difference between prediction and response, not the other way around.) – whuber Mar 20 '12 at 15:56
Single random variables have variances. Residuals are random variables - they are functions of the data. So, single residuals (standardized or not) have variances. – guest Mar 20 '12 at 23:09
#whuber The textbook is "", page,89. It discussed kinds of residuals. ordinary residual is response-prediction. @guest " Single random variables have variances", this is what I dont understand, variables is a property for a sample, isnt it? why single value in a sample (such as a residual) have variance? – hiberbear Mar 21 '12 at 8:08
Does the book have an author...? That usually makes it easier to find. I think that you are getting sample variance and population variance confused. The residual is unknown before the experiment is carried out. The response is random and so is the residual, since it is a function of the response. When we speak of the variance of the residual, we talk about the variance of the underlying random variable. – MånsT Mar 21 '12 at 10:07
sorry for the inconvenience, the authors are SAMPRIT CHATTEFUEE and ALI S. HADI, Regression Analysis by Example, fourth edition. – hiberbear Mar 21 '12 at 15:55

I would say that an individual number (such as a residual), which resulted from a random draw from a probability distribution, is a realized value, not a random variable. Likewise, I would say that the set of $N$ residuals, calculated from your data and your model fit using $\bf{e}=\bf{y}-\bf{\hat{y}}$, is a set of realized values. This set of numbers may be loosely conceptualized as independent draws from an underlying distribution $\epsilon$ ~ $\mathcal{N}(\mu,\sigma^2)$. (Unfortunately however, there are several additional complexities here. For example, you do not actually have $N$ independent pieces of information, because the residuals, $\bf{e}$, must satisfy two conditions: $\sum e_i=0$, and $\sum x_ie_i=0$.)

Now, given some set of numbers, be they residuals or whatever, it is certainly true that they have a variance, $\sum(e_i-\bar{e})^2/N$, but this is uninteresting. What we care about is being able to say something about the data generating process (for instance, to estimate the variance of the population distribution). Using the preceding formula, we could give an approximation by replacing the $N$ with the residual degrees of freedom, but this may not be a good approximation. This is a topic that can get very complicated very fast, but a couple of possible reasons could be heteroscedasticity (i.e., that the variance of the population differs at different levels of $x$), and the presence of outliers (i.e., that a given residual is drawn from a different population entirely). Almost certainly, in practice, you will not be able to estimate the variance of the population from which an outlier was drawn, but nonetheless, in theory, it does have a variance. I suspect something along these lines is what the authors had in mind, however, I should note that I have not read that book.

Update: Upon rereading the question, I suspect the quote may be referring to the way the $x$-value of a point influences the fitted regression line, and thereby the value of the residual associated with that point. The key idea to grasp here is leverage. I discuss these topics in my answer here: Interpreting plot.lm().

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.