# Why does the variance of the Random walk increase?

The random walk that is defined as $$Y_{t} = Y_{t-1} + e_t$$, where $$e_t$$ is white noise. Denotes that the current position is the sum of the previous position + an unpredicted term.

You can prove that the mean function $$\mu_t = 0$$, since

$$E(Y_{t}) = E(e_1+ e_2+ ... +e_t) = E(e_1) + E(e_2) +... +E(e_t) = 0 + 0 + \cdots + 0$$

But why is it that the variance increases linearly with time?

Does this have something to do with it's not “pure” random, since the new position is very correlated with the previous one?

EDIT:

Now I have a much better understanding by visualizing a big sample of random walks, and here we can easily observe that the overall variance does increase over time, and the mean is as expected around zero.

Maybe this was trivial after all, since in the very early stages of the time series (compare time = 10 with time = 100) the random walkers have not had yet the time to explore as much.

• It is hard to see how the "mean" of any one simulated random walk would be the same thing as the expectation of a particular $Y_t$. That expectation is, by definition, computed over the entire "ensemble" of possible random walks, of which your simulated walk is just one instance. When you simulate many walks--perhaps by overlaying their graphs on one plot--you will see that they are spread around the horizontal axis. How does that spread vary with $t$?
– whuber
Jul 2 '15 at 15:19
• @whuber that makes more sense! Ofcourse I should consider it as one instance of all possible walks. And then yes, you can see by looking at the graph that the overall variance of all the walks does increase over time. Thats correct? Jul 2 '15 at 15:27
• Yes, that's right. It's a good way to appreciate what @Glen_b wrote in his answer using mathematics. I have found it helps to be familiar with many applications of random walks: besides the classical Brownian motion application, they describe diffusion, options pricing, the accumulation of measurement errors, and much more. Take one of these, such as diffusion. Imagine a drop of ink falling into a pool of stationary water. Although its position is fixed, it spreads out as time goes by: this is how we can actually see a constantly zero mean together with an increasing variance.
– whuber
Jul 2 '15 at 16:25
• @whuber Thank you so much, I totally understand it now! Jul 2 '15 at 16:39

In short because it keeps adding the variance of the next increments to the variability we have in getting to where we are now.

$$\text{Var}(Y_{t}) = \text{Var}(e_1+ e_2+ ... +e_t)$$
$$\qquad\quad\;\;= \text{Var}(e_1) + \text{Var}(e_2) +... +\text{Var}(e_t)$$ (independence)
$$\qquad\quad\;\;= \sigma^2 + \sigma^2 + ... + \sigma^2=t\sigma^2\,,$$

and we can see that $$t\sigma^2$$ increases linearly with $$t$$.

The mean is zero at each time point; if you simulated the series many times and averaged across series for a given time, that would average to something near 0 $$\quad^{\text{Figure: 500 simulated random walks with sample mean in white and }}$$
$$\quad^{ \pm \text{ one standard deviation in red. Standard deviation increases with } \sqrt{t}\,.}$$

• Yeah, each error term is independet yes. And sure this makes sense on the paper. But I still dont get a good gut feeling for "How can the variance increase linearly" but the mean remains zero? It sounds so wierd, almost like a contradtiction. How about a less mathematical explanation that answers my questions? Jul 2 '15 at 15:22
• timpal0l - At each point in time, you're adding another term that doesn't shift the mean any but adds to the "noise" (the variance about the mean). So the mean stays the same but the variance increases (the distribution "spreads out" more at later times). That's both the intuitive idea and also in a general sense what the mathematics shows. Jul 3 '15 at 0:01
• Thanks for the diagram, A.Webb. Very nice. Jul 3 '15 at 0:11

Here's a way to imagine it. To simplify things, let's replace your white noise $e_i$ with a coin flip $e_i$

$$e_i = \left\{ \begin{array}{c} 1 \ \text{with} \ Pr = .5 \\ -1 \ \text{with} \ Pr = .5 \end{array} \right.$$

this just simplifies the visualization, there's nothing really fundamental about the switch except easing the strain on our imagination.

Now, suppose you have gathered an army of coin flippers. Their instructions are to, at your command, flip their coin, and keep a working tally of what their results were, along with a summation of all their previous results. Each individual flipper is an instance of the random walk

$$W = e_1 + e_2 + \cdots$$

and aggregating over all of your army should give you a take on the expected behavior.

flip 1: About half of your army flips heads, and half flips tails. The expectation of the sum, taken across your whole army, is zero. The maximum value of $W$ across your whole army is $1$ and the minimum is $-1$, so the total range is $2$.

flip 2: About half flip heads, and half flips tails. The expectation of this flip is again zero, so the expectation of $W$ over all flips does not change. Some of your army has flipped $HH$, and some others have flipped $TT$, so the maximum of $W$ is $2$ and the minimum is $-2$; the total range is $4$.

...

flip n: About half flip heads, and half flips tails. The expectation of this flip is again zero, so the expectation of $W$ over all flips does not change, it is still zero. If your army is very large, some very lucky soldiers flipped $HH \cdots H$ and others $TT \cdots T$. That is, there is a few with $n$ heads, and a few with $n$ tails (though this is getting rarer and rarer as time goes on). So, at least in our imaginations, the total range is $2n$.

So here's what you can see from this thought experiment:

• The expectation of the walk is zero, as each step in the walk is balanced.
• The total range of the walk grows linearly with the length of the walk.

To recover intuition we had to discard the standard deviation and use in intuitive measure, the range.

• The standard deviation does not grow linearly, so the final remark is questionable. Jul 2 '15 at 18:01
• Yes, I'm trying to think through something to say to resolve that, any suggestions? All I can think of is appeals to the central limit theorem which are not very intuitive. Jul 2 '15 at 18:01
• @JuhoKokkala I agree with your criticism, so I removed the final remark. Jul 2 '15 at 18:07

Does this have something to do with it's not "pure" random, since the new position is very correlated with the previous one?

It appears that by "pure" you mean independent. In random walk only the steps are random and independent of each other. As you noted, the "positions" are random but correlated, i.e. not independent.

The expectation of the position is still zero like you wrote $E[Y_t]=0$. The reason why you observe non-zero positions is because the positions are still random, i.e. $Y_t$ are all non zero random numbers. As a matter of fact, while you increase the sample larger $Y_t$ will be observed from time to time, precisely because, as you noted, the variance is increasing with sample size.

The variance is increasing because if you unwrap the position as follows: $Y_t=Y_0+\sum_{i=0}^t\varepsilon_t$, you can see that the position is a sum of steps, obviously. The variances add up with sample size increasing.

By the way, the means of errors also add up, but in a random walk we usually assume that the means are zero, so adding all zeros will still result in zero. There's random walk with a drift: $Y_t-Y_{t-1}=\mu+\varepsilon_t$, where $Y_t$ will drift away from zero at rate $\mu t$ with sample time.

Let's take a different example for an intuitive explanation: throwing darts at a dartboard. We have a player, who tries to aim for the bullseye, which we take to be a coordinate called 0. The player throws a few times, and indeed, the mean of his throws is 0, but he's not really good, so the variance is 20 cm.

We ask the player to throw a single new dart. Do you expect it to hit bullseye?

No. Although the mean is exactly bullseye, when we sample a throw, it's quite likely not to be bullseye.

In the same way, with random walk, we don't expect a single sample at time $t$ to be anywhere near 0. That's in fact what the variance indicates: how far away do we expect a sample to be?

However, if we take a lot of samples, we'll see that it does center around 0. Just like our darts player will almost never hit bullseye (large variance), but if he throws a lot of darts, he will have them centered around the bullseye (mean).

If we extend this example to the random walk, we can see that the variance increases with time, even though the mean stays at 0. In the random walk case, it seems strange that the mean stays at 0, even though you will intuitively know that it almost never ends up at the origin exactly. However, the same goes for our darter: we can see that any single dart will almost never hit bullseye with an increasing variance, and yet the darts will form a nice cloud around the bullseye - the mean stays the same: 0.

• This does not describe the phenomenon of the question, which concerns the temporal increase in the spread. That increase is not a function of the number of samples. It is intrinsic.
– whuber
Jul 2 '15 at 16:26
• @whuber I know this answer does not address that, and I had no intention to do so. The OP seemed to struggle with the fact that the mean was completely independent of the variance, even though intuitively we can see that a random walk will almost never end up at the origin, so I tried to clarify that by an example without the difficult dependence on $t$. However, it was too long for a comment, but indeed not intended as a full answer. I extended the answer to hopefully adress your concern a little. Jul 2 '15 at 16:49

Here's another way to get intuition that variance increases linearly with time.

Returns increase linearly with time. $$.1\%$$ return per month translate into $$1.2\%$$ return per year - $$X$$ return per day generate $$365X$$ return per year (assuming independence).

It makes sense that the range of returns also increases linearly. If monthly return is $$.1\%$$ on average $$\pm .05\%$$, then it makes intuitive sense that per year it is $$1.2\%$$ on average $$\pm .6\%$$.

Well, if we intuitively think of variance as range, then it makes intuitive sense that variance increases in the same fashion as return through time, that is linearly.

The answer from Glen B already sufficiently and easily/briefly shows why the variance scales linearly with time.

This answer will give an alternative viewpoint. Possibly one might find the equation for the addition of variance not so intuitive or different; and this answer might give an additional intuition from a different angle.

This alternative viewpoint will consider the probability distribution as a solution to a differential equation that relates to a diffusion process (in particular the Gaussian distribution as a limit distribution). It is related to this question from which the image below is taken.

You can view the random walk as a diffusion process. The following steps give the rough idea

• Considered the probability distribution for the random walk to be at some point $$x$$ at time $$t$$. (consider an ensemble of many walks)

• The probability (density) $$P(x,t)$$ will relate to the probability at some earlier time.

Let's for simplicity use a random walk with discrete steps in discret time. For instance, each time step the random walk takes a step $$\pm 1$$ with equal probability $$p=0.5$$. This is equivalent to taking each two time steps a step $$\pm 2$$ with equal probability $$p=0.25$$, and staying in place with probability $$p=0.5$$. Then

$$P(x,t) = \underbrace{\frac{1}{4} \cdot P(x-1,t-2)}_{\substack{\text{a quarter} \\ \text{from the left}}} + \underbrace{ \frac{1}{2} \cdot P(x,t-2) }_{\substack{\text{half the value} \\ \text{from the spot x}}} + \underbrace{\frac{1}{4} \cdot P(x+1,t-2)}_{\substack{\text{a quarter} \\ \text{from the right}}}$$

• Or in terms of differences

$$\begin{array}{rcl}{\nabla_t}\hphantom{^2} P(x,t) &=& P(x,t)-P(x,t-2) \\ {\nabla_x}^2 P(x,t) &=& [P(x+1,t)-P(x,t)]-[P(x-1,t)-P(x,t)]\end{array}$$

you get

$$\nabla_t P(x,t) = \frac{1}{4} {\nabla_x}^2 P(x,t)$$

• You could view this in the limit as a diffusion process (more specifically the heat equation)

$$\frac{\partial}{\partial t} f(x,t) = D \frac{\partial^2}{\partial x^2} f(x,t)$$

• The normal distribution with $$\sigma \propto \sqrt{t}$$ satisfies the above equation process. 