# Errors and Residuals

In Wikipedia , it is written that :

the sum of the residuals within a random sample is necessarily zero, and thus the residuals are necessarily not independent. The statistical errors on the other hand are independent, and their sum within the random sample is almost surely not zero.

But one of our assumptions are $\mathbb E(\epsilon_i)=0$ . Doesn't it imply $\sum\epsilon_i=0$ . If so , then errors CANNOT also be independent because Wikipedia says that $\sum e_i=0$ implies residuals are not independent .

N.B : $\epsilon$ denotes statistical error while $e$ denotes residual .

• Following this question , another question arises :

In this pdf , in the section of MULTILEVEL ANALYSIS at very beginning it is written that :

The usual assumption is either the sample units themselves or the corresponding RESIDUALS in some statistical model are independently and identically distributed .

But in Wikipedia , they have mentioned RESIDUALS are not independent (i.e., dependent) .

Then how is the assumption "either the sample units themselves or the corresponding RESIDUALS in some statistical model are independently and identically distributed ? "

## 2 Answers

But one of our assumptions are $E(ϵ_i)=0$ . Doesn't it imply $∑ϵ_i=0$

No. Random variables are (generally) not equal to their expectation.

The pdf you quote seems to conflate residual with error. The statement isn't correct.

• That is $E(\varepsilon_i)=\sum_{i=1}^{N}\varepsilon_i P(\varepsilon_i)$ . But (1) Isn't $\sum_{i=1}^{N}\varepsilon_i=0$ ? (2) Aren't residuals also random variable ?
– ABC
Apr 28 '15 at 11:04
• If $\sum_{i=1}^{N}\varepsilon_i\ne 0$ , then why do we minimize $\sum_{i=1}^{N}\varepsilon_i^2$ instaed of $\sum_{i=1}^{N}\varepsilon_i$ in OLS estimation ?
– ABC
Apr 28 '15 at 11:18
• @ABC You can't calculate $\sum_{i=1}^{N}\varepsilon_i^2$ for the simple reason that you can't measure $\varepsilon_i$ - it's the distance from the true regression line and you don't know that (as you don't have access to the entire population), only the fitted regression line (since it's calculated from the sample). Using your notation we minimise $\sum_{i=1}^{N}e_i^2$ Apr 28 '15 at 11:34
• if $\varepsilon_i$ is random , we can't write $\sum_i \varepsilon_i=0$ . Is it ? Then how residuals $\sum_i e_i=0$ ? Aren't residuals random ?
– ABC
Apr 28 '15 at 12:29
• @ABC Good question. The residuals are random but subject to a set of $p+1$ linear constraints $x_j'e=0$, where $j$ indexes a column of the $X$ matrix (matrix of IVs). The "$0$"th column of that is the constant term (a column of $1$'s), which means one constraint is $x_0'e=1'e=\sum_i e_i=0$. The errors exist in a space of dimension $n$ (are unrestricted), while the residuals (while random) are in a subspace of dimension $n-p-1$ where $p$ is the number of IVs. Apr 28 '15 at 15:58

$\varepsilon_i$ is random.

So $\sum \varepsilon_i$ has a distribution instead of being fixed.

$e_i$'s are residuals. They are realizations of ϵi's given that the model is correct.

$\sum e_i = 0$ is guaranteed by the OLS algorithm for regression if you check the formula.

• I improved formatting of your answer. Notice that CV enables $\LaTeX$ formatting - I recommend it for future edits since it improves readability and is less prone of errors due to strange charterer encoding issues.
– Tim
Apr 28 '15 at 6:51
• Are $e_i$'s realizations of $\varepsilon_i$ 's ? But i did know $e_i$'s are estimates of $\varepsilon_i$ 's .
– ABC
Apr 28 '15 at 10:49
• @ABC in conventional notation, no, the $e_i$ is not a realization of $\epsilon_i$ because it's measured from the fitted regression line (which you know, because you can calculate it from your sample), not the true regression line (which you can't measure from, because you don't know the whole population). I think this answer ought to clarify the terminology it is using - it's not true to say $\sum \varepsilon_i = 0$, as Glen_b points out in his answer - if $\epsilon$ is used for the disturbance/error term. Apr 28 '15 at 11:11