Convergence in probability vs. almost sure convergence

Question

I've never really grokked the difference between these two measures of convergence. (Or, in fact, any of the different types of convergence, but I mention these two in particular because of the Weak and Strong Laws of Large Numbers.)

Sure, I can quote the definition of each and give an example where they differ, but I still don't quite get it.

What's a good way to understand the difference? Why is the difference important? Is there a particularly memorable example where they differ?

Answer 1

From my point of view the difference is important, but largely for philosophical reasons. Assume you have some device, that improves with time. So, every time you use the device the probability of it failing is less than before.

Convergence in probability says that the chance of failure goes to zero as the number of usages goes to infinity. So, after using the device a large number of times, you can be very confident of it working correctly, it still might fail, it's just very unlikely.

Convergence almost surely is a bit stronger. It says that the total number of failures is finite. That is, if you count the number of failures as the number of usages goes to infinity, you will get a finite number. The impact of this is as follows: As you use the device more and more, you will, after some finite number of usages, exhaust all failures. From then on the device will work perfectly.

As Srikant points out, you don't actually know when you have exhausted all failures, so from a purely practical point of view, there is not much difference between the two modes of convergence.

However, personally I am very glad that, for example, the strong law of large numbers exists, as opposed to just the weak law. Because now, a scientific experiment to obtain, say, the speed of light, is justified in taking averages. At least in theory, after obtaining enough data, you can get arbitrarily close to the true speed of light. There wont be any failures (however improbable) in the averaging process.

Let me clarify what I mean by ''failures (however improbable) in the averaging process''. Choose some $\delta > 0$ arbitrarily small. You obtain $n$ estimates $X_1,X_2,\dots,X_n$ of the speed of light (or some other quantity) that has some `true' value, say $\mu$. You compute the average $$S_n = \frac{1}{n}\sum_{k=1}^n X_k.$$ As we obtain more data ($n$ increases) we can compute $S_n$ for each $n = 1,2,\dots$. The weak law says (under some assumptions about the $X_n$) that the probability $$P(|S_n - \mu| > \delta) \rightarrow 0$$ as $n$ goes to $\infty$. The strong law says that the number of times that $|S_n - \mu|$ is larger than $\delta$ is finite (with probability 1). That is, if we define the indicator function $I(|S_n - \mu| > \delta)$ that returns one when $|S_n - \mu| > \delta$ and zero otherwise, then $$\sum_{n=1}^{\infty}I(|S_n - \mu| > \delta)$$ converges. This gives you considerable confidence in the value of $S_n$, because it guarantees (i.e. with probability 1) the existence of some finite $n_0$ such that $|S_n - \mu| < \delta$ for all $n > n_0$ (i.e. the average never fails for $n > n_0$). Note that the weak law gives no such guarantee.

Answer 2

I know this question has already been answered (and quite well, in my view), but there was a different question here which had a comment @NRH that mentioned the graphical explanation, and rather than put the pictures there it would seem more fitting to put them here.

So, here goes. It's not as cool as an R package. But it's self-contained and doesn't require a subscription to JSTOR.

In the following we're talking about a simple random walk, $X_{i}= \pm 1$ with equal probability, and we are calculating running averages, $$ \frac{S_{n}}{n} = \frac{1}{n}\sum_{i = 1}^{n}X_{i},\quad n=1,2,\ldots. $$

The SLLN (convergence almost surely) says that we can be 100% sure that this curve stretching off to the right will eventually, at some finite time, fall entirely within the bands forever afterward (to the right).

The R code used to generate this graph is below (plot labels omitted for brevity).

n <- 1000;  m <- 50; e <- 0.05
s <- cumsum(2*(rbinom(n, size=1, prob=0.5) - 0.5))
plot(s/seq.int(n), type = "l", ylim = c(-0.4, 0.4))
abline(h = c(-e,e), lty = 2)

---

The WLLN (convergence in probability) says that a large proportion of the sample paths will be in the bands on the right-hand side, at time $n$ (for the above it looks like around 48 or 9 out of 50). We can never be sure that any particular curve will be inside at any finite time, but looking at the mass of noodles above it'd be a pretty safe bet. The WLLN also says that we can make the proportion of noodles inside as close to 1 as we like by making the plot sufficiently wide.

The R code for the graph follows (again, skipping labels).

x <- matrix(2*(rbinom(n*m, size=1, prob=0.5) - 0.5), ncol = m)
y <- apply(x, 2, function(z) cumsum(z)/seq_along(z))
matplot(y, type = "l", ylim = c(-0.4,0.4))
abline(h = c(-e,e), lty = 2, lwd = 2)

Answer 3

One thing that helped me to grasp the difference is the following equivalence

$$P\left({\lim_{n\to\infty}|X_n-X|=0}\right) = 1 \iff \lim_{n\to\infty} P\left({\sup_{m\geq n}|X_m-X|>\epsilon }\right) = 0 ~\forall \epsilon > 0.$$

In comparison stochastic convergence:

$$\lim_{n\to\infty}P(|X_n-X|>\epsilon) = 0 ~\forall \epsilon >0.$$

When comparing the right side of the upper equivalence with the stochastic convergence, the difference becomes clearer, I think.

Answer 4

I understand it as follows,

Convergence in probability

The probability that the sequence of random variables equals the target value is asymptotically decreasing and approaches 0 but never actually attains 0.

Almost Sure Convergence

The sequence of random variables will equal the target value asymptotically but you cannot predict at what point it will happen.

Almost sure convergence is a stronger condition on the behavior of a sequence of random variables because it states that "something will definitely happen" (we just don't know when). In contrast, convergence in probability states that "while something is likely to happen" the likelihood of "something not happening" decreases asymptotically but never actually reaches 0. (something $\equiv$ a sequence of random variables converging to a particular value).

The wiki has some examples of both which should help clarify the above (in particular see the example of the archer in the context of convergence in prob and the example of the charity in the context of almost sure convergence).

From a practical standpoint, convergence in probability is enough as we do not particularly care about very unlikely events. As an example, consistency of an estimator is essentially convergence in probability. Thus, when using a consistent estimate, we implicitly acknowledge the fact that in large samples there is a very small probability that our estimate is far from the true value. We live with this 'defect' of convergence in probability as we know that asymptotically the probability of the estimator being far from the truth is vanishingly small.

Answer 5

If you enjoy visual explanations, there was a nice 'Teacher's Corner' article on this subject in the American Statistician (cite below). As a bonus, the authors included an R package to facilitate learning.

@article{lafaye09,
  title={Understanding Convergence Concepts: A Visual-Minded and Graphical Simulation-Based Approach},
  author={Lafaye de Micheaux, P. and Liquet, B.},
  journal={The American Statistician},
  volume={63},
  number={2},
  pages={173--178},
  year={2009},
  publisher={ASA}
}