Noise cancels but variance sums - contradiction?

Question

I have been told both things with regard to e.g. summing noisy time series, to justify opposing expectations.

On the one hand, I have been told to expect that summing multiple noisy inputs should lead to noise reduction for the output (the sum), because the noise components average to zero. I imagine this explains why larger sample sizes increase sensitivity, or why brains can generate exquisitely patterned activity despite individual neurons being very noisy. It all cancels out in aggregate.

On the other hand, I have been told to expect that variance sums, so the combination of multiple highly variable inputs should lead to a many times more variable output (N times). This too seems very intuitive, fitting with the idea that "error accumulates" like in dead reckoning.

When faced with the sum of noisy inputs, when should we expect that the noise cancels versus accumulates in the output? What am I missing here that makes these two facts appear contradictory?

Answer 1

I think you've confused averaging, and summing. You're almost onto it with "because the noise components average to zero."

Consider a series of independent variables $\{X_i\}$ (with $i$ running from $1$ to $n$). Let's say they all have expectation $\mu$ and variance $\sigma^2$.

Variance does sum. If we define sum $T=\sum_{i=1}^n X_i$, then by some fairly simple properties of variances of sums, we find the expectation is $n\mu$ and variance $n \sigma^2$. So that aligns with your "I have been told to expect that variance sums" comment. And it does - as $n$ increases, so does the variance of the sum.

Your "the noise components average to zero" correctly reflects means. The mean is $\frac{1}{n} \sum_{i=1}^n X_i = T/n$. Using some algebra, we find that the expectation of this function is $\mu$ and the variance is $\sigma^2/n$. In this case, as $n$ increases, the variance decreases.

In conclusion - there's nothing contradictory in these facts. You just need care and clarity in language.

Answer 2

It depends if you ask about summing or averaging random variables. By the basic properties of variance, for the sum of independent variables, their variance is

$$ \mathrm{Var}\Big(\sum_{i=1}^n X_i\Big) = \sum_{i=1}^n \mathrm{Var}(X_i) $$

If you average them, it however is smaller than the average of variances

$$ \mathrm{Var}\Big(\tfrac{1}{n} \sum_{i=1}^n X_i\Big) = \frac{ \sum_{i=1}^n \mathrm{Var}(X_i)}{n^2} $$

and with $n \to \infty$ the variance goes to zero.

Answer 3

Let $X_i$ be independent variables with mean $\mu$ and variance $\sigma^2$. Then the mean and variance of a sum scale like:

$$\begin{array}{rcl} \text{Mean}\left(\sum_{i=1}^n X_i\right) &=& n\mu\\ \text{Var}\left(\sum_{i=1}^n X_i\right) &=& n\sigma^2 \end{array}$$

So indeed the variance increases and scales like $\propto n$.

But the signal to noise ratio or coefficient of variation does not

$$S/N = \frac{\text{Mean}\left(\sum_{i=1}^n X_i\right)}{\sqrt{\text{Var}\left(\sum_{i=1}^n X_i\right)}} \propto \sqrt{n}$$

$$CV = \frac{\sqrt{\text{Var}\left(\sum_{i=1}^n X_i\right)}}{\text{Mean}\left(\sum_{i=1}^n X_i\right)} \propto \frac{1}{\sqrt{n}}$$

So you get more signal and less (relative) variance.

---

A related question is: Binomial distribution for gender discrimination?

In that question we consider a binomial distribution with increasing sample size (basically a sum of Bernoulli distributed variables). As the mean increase, the standard deviation increases as well, but with a smaller rate.

---

I have been told to expect that variance sums, so the combination of multiple highly variable inputs should lead to a many times more variable output

This may occur in a context of error propagation. For example an experiment where we measure the weight of a liter of milk. Then we have an error in measuring the volume of a liter, and in addition an error in measuring the weight. These two errors add up.

On the one hand, I have been told to expect that summing multiple noisy inputs should lead to noise reduction for the output

This is like explained above. The noise reduction is in a relative sense. When you sum values then the signal scales with $n$ while the noise scales only with $\sqrt{n}$.

Answer 4

what you are missing is that only the variance sums for uncorrelated random variables, but for 100% correlated variables the standard deviation sums.

so if I have n common signals (standard deviation $\tau$) distorted by independent noise (0 mean, standard deviation $\sigma$), $s+\epsilon_i$, then the signal to noise ratio of the sum is $\frac{n^2 \tau^2}{ n \sigma^2}=\frac{n \tau^2}{ \sigma^2}$. ie the signal to noise ratio is $n$ times greater than that of a single distorted signal.

Answer 5

There are many good answers with formulas and plots, but I will add a simple intuitive answer.

Consider summing independent samples of some noisy signal. Yes, the variance does sum. And so the noise does increase. However, the signal also increases because you take a sum of several samples. And it turns out that the signal increases quicker than the noise, so that the relative value of noise to signal, the signal-to-noise ratio, does in fact decrease.

More formally: assume you sum $n$ independent samples. The variance grows proportional to $n$, and so does the value of the signal (the expected value). However, you can not directly compare the variance to the value, because the variance is related to the square of the value, it has different units than the value. You can calculate standard deviation, which is square root of variance, and it can be directly compared to the value of the signal. And then you see that standard deviation grows proportional to $\sqrt{n}$ (because it is the square root of variance, which grows proportional to $n$), while the signal value grows proportional to $n$, that is, much faster. So the signal-to-noise ratio decreases proportional to $\sqrt{n}$.

So much for the sum of samples. You can also take average of the samples, not simple sum. The average is the sum divided by $n$. Then the signal value (the expected value) does not change and does not grow. However, the variance in this case does decrease, because the variance of the sum of $n$ samples grows proportional to $n$, but then we divide the sum by $n$ to take the average, which means that the variance is divided by $n^2$ (remember that variance is related to the square of value). So in the end the variance decreases proportional to $n$, and the standard deviation decreases proportional to $\sqrt{n}$.

Answer 6

There are two key points:

1. The total noise grows in the sum, but it shrinks in the average.
2. Intuitively, noise is measured by standard deviation, not the variance, so the noise grows more slowly than the sum does.

---

Let's flip a fair coin $n$ times and look at the sum (total number of heads):

$n$	95% confidence	standard deviation	variance
$100$	$50 \pm 10$	$5$	$25$
$1000$	$500 \pm 31$	$16$	$250$
$10000$	$5000 \pm 98$	$50$	$2500$
$100000$	$50000 \pm 310$	$158$	$25000$
$1000000$	$500000 \pm 980$	$500$	$250000$

Notice the variance is growing fast, but the noise isn't (however, it is growing).

---

Now let's look at the average:

$n$	95% confidence	standard deviation	variance
$100$	$0.5 \pm 0.1$	$0.05$	$0.0025$
$1000$	$0.5 \pm 0.031$	$0.016$	$0.00025$
$10000$	$0.5 \pm 0.0098$	$0.005$	$0.000025$
$100000$	$0.5 \pm 0.0031$	$0.0016$	$0.0000025$
$1000000$	$0.5 \pm 0.00098$	$0.0005$	$0.00000025$

Answer 7

Here are a couple of charts I did for another purpose (the law of the iterated logarithm) but they may also help here. Both have $100$ different simulations of over $10,000$ cases of independent noise with mean $\mu=0$ and variance $\sigma^2=1$ so standard deviation $\sigma=1$ noise. The $100$ simulations are the same in both charts (see the upper grey line on the left or yellow in the middle), but you get a very different perspective depending on whether you are looking at the averages or the sums.

The first chart shows the cumulated averages. These tend to converge to $0$ as the sample size increases (a law of large numbers result), and the variance of the average is $\frac1n$ so the standard deviation of the average is $\frac1{\sqrt{n}}$, both decreasing with $n$.

The second chart shows the cumulated sums. These do not converge as the sample size increases. The variance of the sum is the sum of the variances so $n$ here and thus the standard deviation of the sum is ${\sqrt{n}}$, both increasing with $n$.