0
$\begingroup$

If I generate a list of elements which has a Gaussian distribution with zero mean:

List = np.random.normal(0, 1, 500)

my intuition (why is obviously wrong) tells me that when we sum all the elements up the result would be close to zero

np.sum(List) ≃ 0

just like the integral of a sine function is zero when we integrate it across one period. Where does my intuition go wrong?

$\endgroup$
2
  • 2
    $\begingroup$ Both the sum & the mean of an iid sample of size $n$ from a normal distribution will have a normal distribution with zero mean. But the variance of the sum will be $n\sigma^2$ and the variance of the mean will be $\sigma^2/n$ where $\sigma^2$ is the variance of the normal distribution being sampled. So the observed sample mean should be closer to zero than the sample sum. $\endgroup$ – Michael R. Chernick Nov 4 '19 at 18:16
  • $\begingroup$ When you toss a fair coin, the chance of a head is 50%. When you toss it 100 times, the chance of exactly 50 heads is not 1. $\endgroup$ – Glen_b Dec 18 '19 at 4:48
3
$\begingroup$

Suppose we are simulating $N$ values from a standard normal distribution, as you do. With some notation, we can write this as $X_1, ..., X_N \sim N(0, 1)$ (iid).

In this case, the expected value of the sum of these is indeed 0: $$E(\sum_{i=1}^{N} X_i) = \sum_{i=1}^{N} E(X_i) = \sum_{i=1}^{N} 0 = 0$$

However, when you add $N$ standard normal independent random variables, the sum of these terms has a variance that is additive: $$Var(\sum_{i=1}^{N} X_i) = \sum_{i=1}^{N} Var(X_i) = \sum_{i=1}^{N} 1 = N$$

So when you add many terms, the expected value is 0, but the variance is extremely large (and gets larger as you add more and more terms). So in any given simulation, you will find that your sum of simulated data points is very different from 0.

One test you can do is to repeat your process of simulating 500 values, say, 10,000 times, and plot a histogram of your simulated sums. You will probably see it centered around 0, and with an estimated variance close to 500.

In Python, this would look like this:

import numpy as np

sum_values = np.empty(10000)
for i in range(10000):
    sim_values = np.random.normal(0, 1, 500)
    sum_values[i] = np.sum(sim_values)

sum_values.mean()
sum_values.var()

$\endgroup$
2
  • $\begingroup$ That is what I also said in my comment except I discussed it for arbitrary values of the variance $\sigma^2$ and not just the value 1. $\endgroup$ – Michael R. Chernick Nov 4 '19 at 18:21
  • 1
    $\begingroup$ In addition to this very nice answer, it might be useful to point OP towards the name of the phenomenon being simulated: it's a random walk. $\endgroup$ – eric_kernfeld Nov 4 '19 at 22:47
2
$\begingroup$

Your intuition isn't too bad! In a certain sense, you should indeed expect np.sum(List)=0. More precisely: the mean of this sum is 0 (since expectation is linear). However, as you note this is often not the case, and the reason is that the variance of this sum is quite large. If you take $n$ samples of IID standard Gaussians (mean 0, variance 1 in your code), the standard deviation is equal to $\sqrt{n}$ (and in the more general case where the Gaussians have variance $\sigma^2$ the standard deviation goes as $\sqrt{n} \sigma$). Using the handy 68-95-99.7 Rule for Gaussian distributions (since the sum of Gaussians is still Gaussian) we can see that, when $n=500$, about $\frac{1}{3}$ of the time the absolute value of np.sum(List) should be greater than $\sqrt{n} = \sqrt{500} = 22$.

To "correct" this behavior, ie. to ensure that the sum stays "close" its mean, we have to bring down its variance by dividing the sum by $n$ (or equivalently multiplying the sum by $\frac{1}{n}$). We are then guaranteed that the scaled sum will stay close to its mean value, in this case 0. This is related to your point that the integral of a sine function is 0 when integrated across a period. Recall that when you integrate a function, you're not just summing its values over a certain region; first you have to "multiply" it by an infinitessimal area $dx$ (speaking roughly). Here you can think about the $\frac{1}{n}$ factor as the "discrete" version of the infinitessimal. This is made rigorous by the field of measure theory, where the factor of $\frac{1}{n}$ is called the "empirical measure"

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.