How the sum of statistical error of a population is not equal to zero?

Question

While searching for difference between error and residual, found this in Wikipedia's article on Errors and residuals

the sum of the residuals within a random sample is necessarily zero, and thus the residuals are necessarily not independent. The statistical errors on the other hand are independent, and their sum within the random sample is almost surely not zero.

Can anyone explain how the sum of statistical error of a population is not equal to zero? And what is that dependent and independent theory.?

P. S. I am very new to statistics. Answers with example will clear my doubt more.

Answer 1

Let's suppose you have a coffee machine which produces coffee cups of 50ml. Now, due to various external conditions and technical imperfections, the quantities are not exactly 50ml. The real quantities effectively produced are around 50ml. Let's say that the errors are normally distributed with zero mean and standard deviation equals 1. This describes our population.

We can consider taking the process of producing quantities of coffee modeled as $y = \mu + \epsilon$, where $\mu=50$ and $\epsilon \sim Normal(0,1)$. Consider now taking samples, basically, you will collect a number of samples, which can be considered as observing a number of random variables. Let further suppose your observed sample values are $\{y_1, y_2, y_3, y_4, y_5\} = \{49, 51.1, 47.8, 52.1, 50.5\}$.

Now here comes the trick. In reality, because your random variables are modeled as $Y_i = \mu + \epsilon_i$, you will have the following statistical errors $\epsilon_i = \{-1, 1.1, -2.2, 2.1, 0.5\}$. But this knowledge is not available to you. We can ay the following about the statistical errors:

• the statistical errors are unknown to you, you know only the outcomes $y_i$
• the sum of the errors has zero probability to be equal with $0$. I is intuitive why. As an example think about the situation when you take only one sample, which is the probability to produce exactly 50ml?
• on the other hand if the sample is large, the sum of the errors will go closer to zero, even if it will have zero probability t be exactly $0$.

What does a statistician? He is correctly assuming that the random variables are distributed as $Normal(\mu, \sigma^2)$ an he ask what is the value of $\mu$. What he has is the sample observations $y_i$. He wisely thinks that a good estimator of $\mu$ would be the sample mean: $$\hat{\mu} = \frac{1}{n}\sum_{i=1}^{n}y_i$$ This is actually a really good estimator, unbiased, efficient and so on. So he computes it and he get $\hat{\mu}=50.1$. Notice that the estimated values is different than the real unknown value $\mu = 50$. This is normal and it happens for the same reason why the sum of errors has zero probability to be equal with $0$. Intuitively you can think that the errors which happens produce errors in estimation.

The residual errors can now be computed as $$e_i = y_i - \hat{\mu} = \{-1.1, 1.0, -2.3, 2.0, 0.4\}$$.

You can check now and see that the sum of residual errors is $0$. Why is that? It is because of the way the estimator is build. Let's see:

$$\sum_{i=1}^{n} e_i = \sum_{i=1}^{n} (y_i - \hat{\mu}) = \sum_{i=1}^{n}y_i - n \hat{\mu} = \sum_{i=1}^{n}y_i - n \frac{1}{n}\sum_{i=1}^{n}y_i = 0$$

So, by the way we constructed the esimator the residuals are zero. Because the sum is $0$ we state that the residuals are not independent. This can be checked simply by noticing that if you knew the values of the first $n-1$ residuals, than the value of the last residual is precisely the negative sum of the previous residuals. The last residual does not have a chance to be something else, so it is not independent, its value depends on others.

Now another trick. This is always true? Well, this is not always true, and I think the statement from wikipedia that the sum of residuals is zero is not correct, in the sense that it does not state that this happens in that specific context.

Consider the following procedure. The statistician which estimates the mean $\mu$ does not use the sample mean, but sample median. It is still a good estimator, not as efficient as sample mean, but with enough samples it is robust and work as well. Because of the way the sample median is computed, the sum of the residuals will NOT be zero.

So the conclusion is that the statement: Note that the sum of the residuals within a random sample is necessarily zero, and thus the residuals are necessarily not independent. is valid only for this specific case, of using sample mean to estimate the real mean. Of course, it is valid also in other situations, depending on the way how estimators are formulated. The point is that it is not always true.

Answer 2

Note: I have more to say about this, but right now I am struggling to think of ways to explain it clearly, and would like to pause so you can give me feedback about what parts of this answer do or do not make sense to you, so I can change the parts currently within the answer, and plan to better write the remaining parts of the answer so you hopefully understand them.

Right now this is not a complete answer nor does it directly address what you are asking, so it should under no circumstances be accepted in its current form.

---

Part of the reason why the explanation in the Wikipedia article is confusing is because the word "mean" is used to mean (pun unintended) too many different things.

The main difference, when thinking about any of these related concepts, to always keep in mind, is the difference between a population and a sample. (See here, although I don't really like the answers given to the question.)

This also comes down to the difference between a random variable and an observation/realization of a random variable.

That might sound too obvious to be helpful, but it is what always anchored me and helped me to figure things out when I got confused. There are many concepts in statistics which seem similar to each other or somehow redundant, but which one investigates closer, it turns out that a fundamental difference includes the difference between a population and a sample. Whenever you get confused between two concepts in statistics, try asking yourself: "how does this relate to the difference between a population and a sample (from that population)?" Answering that question may anchor you as it has anchored me.

I don't know if the language of probability theory and random variables will either help you or make you more confused, so with that caveat I will continue.

population: In a model of a real-life situation, the mathematical notion of a sample space is what is used to represent a population. Thus the "population mean" corresponds to the notion of expectation of a random variable.

(finite) sample: This is a set of $n$ "realizations" of the random variable. In other words, given a random variable $X$ whose sample space $\Omega$ is the population in question, a sample will be an element of the space $\Omega^n = \Omega \times \dots \times \Omega$, i.e. an $n$-tuple of elements from $\Omega$.

In other words, the population is the entire space $\Omega$, while an individual sample is just one element of that space. The same way that $x \in \mathbb{R}$ and the entire set $\mathbb{R}$ are different kinds of objects.

This is already tricky, but it gets even trickier when one considers sampling. If one "takes a sample at random", then one is observing the outcomes of the random variable $X$, which often causes the distinction between population and sample to appear blurry in theoretical treatments.

If one considers "the" sample, or "a given" sample, then one is talking about a specific element of $\Omega$ ($n = 1$), but if one is talking about "a" sample, then one is talking about a random variable which takes values in the sample space (i.e. the population) $\Omega$.

It becomes even more confusing when one considers (and only knows about) unbiased estimators -- the difference can still be clear when talking about "a given" or "the" sample, but when talking about "a" sample, then one can only talk about the expected value of the estimator (not a particular realization of it) but for unbiased estimators (like the sample mean) the expected value of the estimator is the same as the parameter it is supposed to be estimating, so it is more difficult to see the difference.

Another useful concept to keep in mind here is statistic (Wikipedia also fails us here), which is just any arbitrary function from $\Omega$. But as I mentioned before, understanding this concept well, as well as understanding basically any other concept well in statistics, depends on a clear understanding of the difference between a sample and a population, and this is difficult to explain, since the word sample is also used to mean too many different things (e.g. in particular both the realizations of a random variable as well as the random variable itself). The distinction is usually inferred from the context, but since it is never made explicit, it is difficult to learn in the first place that the distinction exists and needs to be thought about.

Answer 3

Given a set of data, we might want to split it into two portions - one for fitting and obtaining the model (training data) and one for testing the model (testing data). (There are other methods such as cross-validation)

The rationale is it is often not possible to get the entire population, and the model that we obtain might be better fitted within the existing data set (in your case, the random sample) but may not be able to predict beyond the existing data set.

When used on the training data, we call it residual. It is necessarily zero in cases where our best fitted model is computed by minimising the sum of residuals. However, when your model is tested on the testing data (which was not used to obtain the model), we call the "residuals" as "errors" and these are independent. Because of the "randomness", it is unlikely that you will have zero error.