28
$\begingroup$

The random walk that is defined as $Y_{t} = Y_{t-1} + e_t$, where $e_t$ is white noise. Denotes that the current position is the sum of the previous position + an unpredicted term.

You can prove that the mean function $\mu_t = 0 $, since $E(Y_{t}) = E(e_1+ e_2+ ... +e_t) = E(e_1) + E(e_2) +... +E(e_t) = 0 + 0 + ... + 0$

But why is it that the variance increases linearly with time?

Does this have something to do with it's not "pure" random, since the new position is very correlated with the previous one?

EDIT:

Now I have a much better understanding by visualizing a big sample of random walks, and here we can easily observe that the overall variance does increase over time,

100 000 Random walks

and the mean is as expected around zero.

Maybe this was trivial after all, since in the very early stages of the time series (compare time = 10, with 100) the random walkers have not had the time yet to explore as much.

$\endgroup$
  • 2
    $\begingroup$ It is hard to see how the "mean" of any one simulated random walk would be the same thing as the expectation of a particular $Y_t$. That expectation is, by definition, computed over the entire "ensemble" of possible random walks, of which your simulated walk is just one instance. When you simulate many walks--perhaps by overlaying their graphs on one plot--you will see that they are spread around the horizontal axis. How does that spread vary with $t$? $\endgroup$ – whuber Jul 2 '15 at 15:19
  • $\begingroup$ @whuber that makes more sense! Ofcourse I should consider it as one instance of all possible walks. And then yes, you can see by looking at the graph that the overall variance of all the walks does increase over time. Thats correct? $\endgroup$ – Isbister Jul 2 '15 at 15:27
  • 1
    $\begingroup$ Yes, that's right. It's a good way to appreciate what @Glen_b wrote in his answer using mathematics. I have found it helps to be familiar with many applications of random walks: besides the classical Brownian motion application, they describe diffusion, options pricing, the accumulation of measurement errors, and much more. Take one of these, such as diffusion. Imagine a drop of ink falling into a pool of stationary water. Although its position is fixed, it spreads out as time goes by: this is how we can actually see a constantly zero mean together with an increasing variance. $\endgroup$ – whuber Jul 2 '15 at 16:25
  • $\begingroup$ @whuber Thank you so much, I totally understand it now! $\endgroup$ – Isbister Jul 2 '15 at 16:39
37
$\begingroup$

In short because it keeps adding the variance of the next increments to the variability we have in getting to where we are now.

$\text{Var}(Y_{t}) = \text{Var}(e_1+ e_2+ ... +e_t)$
$\qquad\quad\;\;= \text{Var}(e_1) + \text{Var}(e_2) +... +\text{Var}(e_t)$ (independence)
$\qquad\quad\;\;= \sigma^2 + \sigma^2 + ... + \sigma^2=t\sigma^2\,,$

and we can see that $t\sigma^2$ increases linearly with $t$.


The mean is zero at each time point; if you simulated the series many times and averaged across series for a given time, that would average to something near 0

500 simulated random walks with sample mean and +/- standard deviation

$\quad^{\text{Figure: 500 simulated random walks with sample mean in white and }}$
$\quad^{ \pm \text{ one standard deviation in red. Standard deviation increases with } \sqrt{t}\,.}$

$\endgroup$
  • $\begingroup$ Yeah, each error term is independet yes. And sure this makes sense on the paper. But I still dont get a good gut feeling for "How can the variance increase linearly" but the mean remains zero? It sounds so wierd, almost like a contradtiction. How about a less mathematical explanation that answers my questions? $\endgroup$ – Isbister Jul 2 '15 at 15:22
  • $\begingroup$ timpal0l - At each point in time, you're adding another term that doesn't shift the mean any but adds to the "noise" (the variance about the mean). So the mean stays the same but the variance increases (the distribution "spreads out" more at later times). That's both the intuitive idea and also in a general sense what the mathematics shows. $\endgroup$ – Glen_b Jul 3 '15 at 0:01
  • 1
    $\begingroup$ Thanks for the diagram, A.Webb. Very nice. $\endgroup$ – Glen_b Jul 3 '15 at 0:11
15
$\begingroup$

Here's a way to imagine it. To simplify things, let's replace your white noise $e_i$ with a coin flip $e_i$

$$ e_i = \left\{ \begin{array}{c} 1 \ \text{with} \ Pr = .5 \\ -1 \ \text{with} \ Pr = .5 \end{array} \right. $$

this just simplifies the visualization, there's nothing really fundamental about the switch except easing the strain on our imagination.

Now, suppose you have gathered an army of coin flippers. Their instructions are to, at your command, flip their coin, and keep a working tally of what their results were, along with a summation of all their previous results. Each individual flipper is an instance of the random walk

$$ W = e_1 + e_2 + \cdots $$

and aggregating over all of your army should give you a take on the expected behavior.

flip 1: About half of your army flips heads, and half flips tails. The expectation of the sum, taken across your whole army, is zero. The maximum value of $W$ across your whole army is $1$ and the minimum is $-1$, so the total range is $2$.

flip 2: About half flip heads, and half flips tails. The expectation of this flip is again zero, so the expectation of $W$ over all flips does not change. Some of your army has flipped $HH$, and some others have flipped $TT$, so the maximum of $W$ is $2$ and the minimum is $-2$; the total range is $4$.

...

flip n: About half flip heads, and half flips tails. The expectation of this flip is again zero, so the expectation of $W$ over all flips does not change, it is still zero. If your army is very large, some very lucky soldiers flipped $HH \cdots H$ and others $TT \cdots T$. That is, there is a few with $n$ heads, and a few with $n$ tails (though this is getting rarer and rarer as time goes on). So, at least in our imaginations, the total range is $2n$.

So here's what you can see from this thought experiment:

  • The expectation of the walk is zero, as each step in the walk is balanced.
  • The total range of the walk grows linearly with the length of the walk.

To recover intuition we had to discard the standard deviation and use in intuitive measure, the range.

$\endgroup$
  • 1
    $\begingroup$ The standard deviation does not grow linearly, so the final remark is questionable. $\endgroup$ – Juho Kokkala Jul 2 '15 at 18:01
  • $\begingroup$ Yes, I'm trying to think through something to say to resolve that, any suggestions? All I can think of is appeals to the central limit theorem which are not very intuitive. $\endgroup$ – Matthew Drury Jul 2 '15 at 18:01
  • $\begingroup$ @JuhoKokkala I agree with your criticism, so I removed the final remark. $\endgroup$ – Matthew Drury Jul 2 '15 at 18:07
3
$\begingroup$

Does this have something to do with it's not "pure" random, since the new position is very correlated with the previous one?

It appears that by "pure" you mean independent. In random walk only the steps are random and independent of each other. As you noted, the "positions" are random but correlated, i.e. not independent.

The expectation of the position is still zero like you wrote $E[Y_t]=0$. The reason why you observe non-zero positions is because the positions are still random, i.e. $Y_t$ are all non zero random numbers. As a matter of fact, while you increase the sample larger $Y_t$ will be observed from time to time, precisely because, as you noted, the variance is increasing with sample size.

The variance is increasing because if you unwrap the position as follows: $Y_t=Y_0+\sum_{i=0}^t\varepsilon_t$, you can see that the position is a sum of steps, obviously. The variances add up with sample size increasing.

By the way, the means of errors also add up, but in a random walk we usually assume that the means are zero, so adding all zeros will still result in zero. There's random walk with a drift: $Y_t-Y_{t-1}=\mu+\varepsilon_t$, where $Y_t$ will drift away from zero at rate $\mu t$ with sample time.

$\endgroup$
2
$\begingroup$

Let's take a different example for an intuitive explanation: throwing darts at a dartboard. We have a player, who tries to aim for the bullseye, which we take to be a coordinate called 0. The player throws a few times, and indeed, the mean of his throws is 0, but he's not really good, so the variance is 20 cm.

We ask the player to throw a single new dart. Do you expect it to hit bullseye?

No. Although the mean is exactly bullseye, when we sample a throw, it's quite likely not to be bullseye.

In the same way, with random walk, we don't expect a single sample at time $t$ to be anywhere near 0. That's in fact what the variance indicates: how far away do we expect a sample to be?

However, if we take a lot of samples, we'll see that it does center around 0. Just like our darts player will almost never hit bullseye (large variance), but if he throws a lot of darts, he will have them centered around the bullseye (mean).

If we extend this example to the random walk, we can see that the variance increases with time, even though the mean stays at 0. In the random walk case, it seems strange that the mean stays at 0, even though you will intuitively know that it almost never ends up at the origin exactly. However, the same goes for our darter: we can see that any single dart will almost never hit bullseye with an increasing variance, and yet the darts will form a nice cloud around the bullseye - the mean stays the same: 0.

$\endgroup$
  • 1
    $\begingroup$ This does not describe the phenomenon of the question, which concerns the temporal increase in the spread. That increase is not a function of the number of samples. It is intrinsic. $\endgroup$ – whuber Jul 2 '15 at 16:26
  • 1
    $\begingroup$ @whuber I know this answer does not address that, and I had no intention to do so. The OP seemed to struggle with the fact that the mean was completely independent of the variance, even though intuitively we can see that a random walk will almost never end up at the origin, so I tried to clarify that by an example without the difficult dependence on $t$. However, it was too long for a comment, but indeed not intended as a full answer. I extended the answer to hopefully adress your concern a little. $\endgroup$ – Sanchises Jul 2 '15 at 16:49
0
$\begingroup$

Here's another way to get intuition that variance increases linearly with time.

Returns increase linearly with time. $.1\%$ return per month translate into $1.2\%$ return per year - $X$ return per day generate $365X$ return per year (assuming independence).

It makes sense that the range of returns also increases linearly. If monthly return is $.1\%$ on average $\pm .05\%$, then it makes intuitive sense that per year it is $1.2\%$ on average $\pm .6\%$.

Well, if we intuitively think of variance as range, then it makes intuitive sense that variance increases in the same fashion as return through time, that is linearly.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.