What is Epsilon Convergence in Probability?

Question

I understand that the formula for probability of convergence is $P[|X_n − X_\infty| \gt \epsilon ]\to 0$ and I can solve problems using the formula. Can anyone explain it intuitively (like I am a five year old), particularly in regards to what $\epsilon$ is?

Answer 1

Since we're talking about convergence - specifically, in this case, $X_n$ converging to $X_\infty$ - we want to show that $X_n$ gets really, really, really close to $X_\infty$ as $n$ gets larger and larger.

Think of $\varepsilon$ as any really small positive number; say you think $\varepsilon = 0.01$ is good enough. Then in order to show that $X_n$ is really, really, really close to $X_\infty$, we want to show that $X_n$ falls inside $(X_\infty-0.01,X_\infty+0.01)$ for sufficiently large $n$. (Sufficiently large $n$ just means that there is some $n'$ such that for every $n > n'$, $X_n$ is within plus or minus $0.01$ of $X_\infty$ with probability 1.)

But say that I'm not convinced that $X_n$ converges to $X_\infty$ because $\varepsilon=0.01$ just seems too big for me. So instead, let $\varepsilon = 0.0001$. Then I'm convinced that $X_n$ converges to $X_\infty$ (or that $X_n$ is really, really, really close to $X_\infty$) if we can show that, for sufficiently large $n$, $X_n$ falls inside $(X_\infty-0.0001,X_\infty+0.0001)$.

Suppose you have a lot of friends who pick $\varepsilon$ to be smaller and smaller. The idea behind convergence is that for any $\varepsilon > 0$, no matter how small $\varepsilon$ gets, showing that $X_n$ falls inside $X_\infty\pm\varepsilon$ for sufficiently large $n$ demonstrates that $X_n$ converges to $X_\infty$.

In the most basic terms, $\varepsilon$ is just a small positive number. As it relates to convergence, you want to be able to show that for any $\varepsilon > 0$ (so that all of your infinite friends with different $\varepsilon$ values are convinced), the sequence that converges will, at some point, get within plus or minus $\varepsilon$ of the limit to which you believe the sequence converges. If you cannot show that your sequence falls within $\varepsilon$ of the believed limit for some $\varepsilon$, then the sequence cannot converge to that limit.

Answer 2

Sequences of random variables.

Intuition comes from metaphors. The following metaphor, which models random quantities by pulling slips of paper out of a container, captures all the essential mathematical elements while glossing over a technical condition ("measurability") needed to make sense of situations with uncountably many tickets.

---

Consider a tickets-in-a-box model of a sample space $\Omega$: the name of each element $\omega\in\Omega$ is written on a slip of paper (a "ticket") which is put into the box. Elements with greater probability are named on more tickets.

A random variable $X$ is a consistent way of writing a number on each ticket. "Consistent" means that all the tickets for any particular $\omega$ all get the same value of $X$, written $X(\omega)$.

A sequence of random variables $X_1, X_2, \ldots, X_n, \ldots$ therefore can be conceived of as a sequence $X_1(\omega), X_2(\omega),\ldots$ written on each ticket (again in a consistent way).

$X_\infty$ is another random variable, which is one more number written on each ticket.

Events and probability.

Let $\epsilon$ be any real number. We will say more about it below.

The event $|X_n - X_\infty| \ge \epsilon$ describes all the tickets $\omega\in\Omega$ for which the values $X_n(\omega)$ and $X_\infty(\omega)$ differ by $\epsilon$ or more. It's a subset of the tickets in the box. These tickets form some proportion of the box: that proportion models their probability, $\Pr\left(|X_n - X_\infty| \ge \epsilon\right)$.

Limits.

Every assertion about a limit is a form of mathematical game. When we write that some sequence has a limit $L$, what we mean is we can play a game against a hypothetical opponent (who is doing their best to make us lose) and we will always win. In the limit game, your opponent names some positive number--usually a tiny one--which we will call $\delta$. You win if you can remove a finite number of elements from that sequence and show that all the remaining elements are within a distance $\delta$ of $L$. As in any game, you may calibrate your response to your opponent's move: the elements you remove are allowed to depend on $\delta$.

Limits in probability.

Let's apply the limit game to the assertion $\Pr\left(|X_n - X_\infty| \ge \epsilon\right)\to 0$. Because this assertion involves an unspecified quantity $\epsilon$, your opponent may also specify its value. That makes the game as difficult as possible for you to win.

So, no matter what values of $\epsilon$ and $\delta \gt 0$ your opponent specifies, your response will be to cross out some finite number of the random variables $X_i$ on the tickets. For every remaining random variable $X_n$, let the tickets where $X_n(\omega)$ differs from $X_\infty(\omega)$ by $\epsilon$ or more be the "bad" ones for $n$. You win the game provided the proportions of bad tickets are always less than $\delta$ (for all the $X_n$ that remain).

A little thought reveals the subtlety of this game: the bad tickets for $n$ do not have to have any relationship to the bad tickets for $m$ (where $n$ and $m$ designate any of the remaining random variables you didn't cross out). In other words, on any given ticket the values $X_n(\omega)$ can bounce all over the place. The limit in probability is a statement about what's written on all the tickets in the box but it is not a statement about what might be written on any individual ticket.