8
$\begingroup$
  • The sum of the residuals in a linear model equals zero.
  • The residuals in a linear model are independent.

Isn't it a contradiction?

$\endgroup$
2
  • 1
    $\begingroup$ You confuse two meanings of "residual." The first bullet refers to the differences between the data and their fitted values. The second refers to a collection of random variables that are used to model the differences between the data and their expectations. $\endgroup$
    – whuber
    Jul 12 '16 at 13:25
  • 1
    $\begingroup$ The second thing are more often called errors (which has the benefit of clearly distinguishing them from the sample quantities in your first statement). Errors (population quantities) are independent, but residuals (sample quantities) are not. $\endgroup$
    – Glen_b
    Jul 12 '16 at 14:43
11
$\begingroup$

The question appears to confuse two meanings of "residual."

  • The first bullet refers to the differences between the data and their fitted values.

  • The second bullet refers to a collection of random variables that are used to model the differences between the data and their expectations.

This might become clearer upon examining the simplest possible example: estimating the mean of a population, $\mu$, by taking two independent observations from it (with replacement). The data can be modeled by an ordered pair of random variables $(X_1, X_2)$. The "fitted values" are the estimated mean,

$$\bar X = (X_1 + X_2)/2.$$

This number is the fit for each of the two observations.

  • The residuals are the differences between the data and the fit. They consist of the ordered pair $$(e_1, e_2) = (X_1 - \bar X, X_2 - \bar X) = ((X_1-X_2)/2, -(X_1-X_2)/2).$$ Consequently $e_2 = -e_1$, showing the residuals are dependent.

  • An alternative model of these data uses the random variables $$(\epsilon_1, \epsilon_2) = (X_1 - \mu, X_2 - \mu).$$ Often these random variables are called "errors" but sometimes they are also called "residuals." Since the $X_i$ are independent, and $\mu$ is just some constant, the $\epsilon_i$ are also independent.

It might be of interest to note that $e_1 + e_2 = 0$ whereas $\mathbb{E}(\epsilon_1) = \mathbb{E}(\epsilon_2) = 0$. The former is a true dependence among random variables whereas the latter is merely a constraint concerning the underlying model.

$\endgroup$
9
$\begingroup$

First, let's clarify the terminology, which can be different in different fields. For instance, in econometrics we differentiate between errors and residuals. Let's look at a simple model: $$y_i=\beta_0+\beta_1 x_i+\varepsilon_i$$ Here, $\varepsilon_i$ is called errors. They are not observable, i.e. unknowns. The parameters (betas, coefficients) of the model are also unknown.

We can try to fit and estimate the model, and obtain the parameter estimates $\hat\beta_0,\hat\beta_1$, then we can obtain the residuals: $$\hat\varepsilon_i=y-\hat\beta_1 x -\hat\beta_0$$

So, residuals $\hat\varepsilon_i$ are estimates of unobserved errors $\varepsilon_i$.

Why is this so important? Because you mixed and matched several concepts from different places in your question.

The first question alludes to the technical property of linear models. When you estimate the model, $\hat\beta_0$ will absorb the mean of errors making the residual mean zero.

The second question sounds like one of the assumptions of Gauss-Markov theorem, but misplacing the errors with residuals. The theorem's assumption is about errors. It may or may not hold true. Both residuals and errors may show autocorrelation, for instance.

$\endgroup$
1
  • $\begingroup$ +1 The edit gets this answer just right, so I have deleted my previous comment. $\endgroup$
    – whuber
    Jul 12 '16 at 14:13
4
$\begingroup$

The residuals are certainly not independent. Assume that the true errors $u_i, i=1,...,n$ are fully independent. In a linear model

$$y_i = \mathbb x_i'\beta + u_i \implies u_i = y_i - \mathbb x_i'\beta$$

the residuals equal, under OLS estimation

$$\hat u_i = y_i - \mathbb x_i'\hat \beta(\mathbf y, \mathbf X) = y_i - \mathbb x_i' \left(\mathbf X' \mathbf X\right)^{-1}\mathbf X'\mathbf y $$

$$= y_i - \mathbb x_i' \left(\mathbf X' \mathbf X\right)^{-1}\mathbf X'\left (\mathbf X \beta + \mathbf u\right) $$

$$\implies \hat u_i = u_i -\mathbb x_i' \left(\mathbf X' \mathbf X\right)^{-1}\mathbf X'\mathbf u$$

Considering

$$E(\hat u_i \hat u_j) = E(u_iu_j) - E\left[u_i\mathbb x_j' \left(\mathbf X' \mathbf X\right)^{-1}\mathbf X'\mathbf u\right]\\-E\left[u_j\mathbb x_i' \left(\mathbf X' \mathbf X\right)^{-1}\mathbf X'\mathbf u\right] + E\left[\mathbb x_i' \left(\mathbf X' \mathbf X\right)^{-1}\mathbf X'\mathbf u\mathbf u'\mathbf X \left(\mathbf X' \mathbf X\right)^{-1}\mathbb x_j'\right]$$

we can see that it is not equal to zero, because for example,

$$ E\left[u_i\mathbb x_j' \left(\mathbf X' \mathbf X\right)^{-1}\mathbf X'\mathbf u\right] \neq 0$$

since $u_i$ exists in $\mathbf u$ also, and so we get $u_i^2$ whose conditional or unconditional expected value cannot be eqaul to zero (why?).

So the residuals are not independent, even if the true errors are, and even if the regressors are independent from the true errors.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.