# Weak law vs strong law of large numbers - intuition

I was wondering if my intuition behind the weak law (WLLN) and strong law of large numbers (SLLN) is correct.

The WLLN says that, if you consider a sequence $$X_1,X_2,...$$,of $$i.i.d.$$ random variables with finite mean, $$E|X|<\infty$$, then define $$\bar{X}_n=\frac{X_1+X_2+...+X_n}{n}$$.

As $$n \rightarrow \infty$$, the WLLN says that $$\bar{X}_n\xrightarrow{p} E[X]=\mu$$

The intution is that for $$n$$ larger and larger there is higher probability that $$\bar{X}_n$$ takes value closer and closer to $$\mu$$. However, it is still possible that for any $$\epsilon>0$$ , we have $$|\bar{X}_n-\mu|>\epsilon$$ occurs an infinite number of times although at infrequent intervals.

The SLLN claims that as $$n \rightarrow \infty$$, than $$\bar{X}_n \xrightarrow{a.s.} E[X]=\mu$$

However, we can still have $$|\bar{X}_n-\mu|>\epsilon$$, but this will happen a finite number of times.

Are my statements correct?

• I would add that while $|\bar X_n-\mu|>\epsilon$ will only happen a finite number of times for any particular realisation of the sequence (any sample point $\omega\in\Omega$), and so will happen a last time at say, $n=N$, there isn't necessarily a last time across the different realisations of the sequence Commented Feb 9, 2023 at 20:57
• Thank you very much for your comment. Are you referring to the a.s. convergence? Commented Feb 9, 2023 at 21:10
• Commented Feb 9, 2023 at 21:26
• Commented Feb 9, 2023 at 21:47
• there's not much difference between finite and infinite "number of times" because both will have measure zero, as power of continuum of infinite countable sets is still zero Commented Feb 10, 2023 at 19:28

After a second read of your question, I found that there are two pairs of concepts need clarification. First, the an event happens vs. the probability of an event happens. Second, convergence in probability vs convergence almost surely.

### 1. $$A$$ happens vs. The probability of $$A$$ happens

Let $$(\Omega, \mathscr{F}, P)$$ be a probability space and let $$A$$ be an event (i.e., $$A \in \mathscr{F}$$), your post seems to imply the following logic:

If $$P(A) > 0$$ (even when $$P(A)$$ is very small), then $$A$$ happens.

This is a conceptual misunderstanding, which complements another common misunderstanding as follows:

If $$P(A) = 0$$, then $$A$$ cannot happen.

Both logics are wrong in that whether $$A$$ happens is random and cannot be inferred from the magnitude of $$P(A)$$. Instead, whether $$A$$ happens depends on every specific outcome of the underlying experiment or observation, which is usually denoted by $$\omega$$. Therefore, technically speaking, whether $$A$$ happens or not is a binary random variable $$\Omega \to \{0, 1\}$$, defined by $$I_A(\omega)$$, meaning if $$\omega \in A$$, then $$A$$ happens, otherwise $$A$$ does not happen. To put it in another way, knowing whether $$A$$ happens is "outcome-wise" -- without knowing information of a specific outcome $$\omega$$, whether $$A$$ happens is inconclusive.

By contrast, the probability of $$A$$ happens, denoted by $$P(A)$$, is an overall deterministic measurement of the likelihood that $$A$$ happens, which is assigned (or can be derived based on the axioms of probability measure) by the experimenter in advance. In abstract sense, once the probability space has been set up and the event $$A$$ of interest is picked up, then $$P(A)$$ is certainly known. In particular, $$P(A)$$ does not depend on any specific outcome $$\omega$$. By the same token, knowing $$P(A)$$ does not convey any message of a specific outcome $$\omega$$, i.e., whether $$A$$ happens or not.

To illustrate, consider the classical probability space $$((0, 1], \mathscr{B}, \lambda)$$, where $$\mathscr{B}$$ is Borel $$\sigma$$-field on $$(0, 1]$$ and $$\lambda$$ is Lebesgue measure. This probability space can be used to model the observational experiment that if a radioactive substance has emitted a single $$\alpha$$-particle during a unit interval of time, for example.

First consider the event $$A = (0, 1/2]$$. Then, by the setup of this probability space, the probability that $$A$$ happens (i.e., the emission occurs in the time interval $$(0, 1/2]$$) is $$P(A) = \lambda((0, 1/2]) = 1/2 > 0$$. However, only with $$P(A) = 1/2$$, nothing can be said on "whether $$A$$ happens" unless the specific emission instant $$\omega$$ has been observed: if $$\omega = 0.25$$, then $$A$$ happens; if $$\omega = 0.6$$, then $$A$$ does not happen -- so it is completely random. And no matter what $$\omega$$ was actually observed, the deterministic probability $$P(A)$$ does not change.

Next consider the event $$A = \mathbb{Q} \cap (0, 1]$$. Again, by the setup of the probability space, we have $$P(A) = 0$$. However, this does not imply $$A$$ never happens. What determines whether $$A$$ happens is again the realized emission time: if $$\omega = 0.5$$, then $$A$$ happens; if $$\omega = \sqrt{2}/2$$, then $$A$$ does not happen.

Final word: although the distinction between "$$A$$ happens" and "$$P(A)$$" is subtle, in probability and statistics, it is only the "$$P(A)$$" that is of the primary interest. On the other hand, it is very often to see statements "$$A$$ happens with probability $$1$$", which is actually a rephrase of "$$P(A) = 1$$".

### 2. Convergence in probability vs. Convergence almost surely

For generality, we can discuss it for any sequence of random variables $$\{Y_n\}$$ and a limit $$Y$$ and treat $$\bar{X}_n$$ and $$\mu$$ as a special case.

$$Y_n$$ converges to $$Y$$ in probability means: for any $$\epsilon > 0$$, \begin{align} \lim_{n \to \infty}P[|Y_n - Y| > \epsilon] = 0. \tag{1} \end{align} While $$Y_n$$ converges to $$Y$$ almost surely (or "with probability $$1$$") means: for any $$\epsilon > 0$$, \begin{align} P[|Y_n - Y| > \epsilon \;\text{ i.o.}] = 0. \tag{2} \end{align} The notation "i.o." means "infinitely often" (to be elaborated later).

While $$(1)$$ is the original definition of convergence in probability, $$(2)$$ is an equivalent statement of the more primitive definition of almost surely convergence: $$P\left[\lim\limits_{n \to \infty}Y_n = Y\right] = 1$$ or $$P\left[\left[\lim\limits_{n \to \infty}Y_n = Y\right]^c\right] = 0$$ (for a proof of the equivalence, see Probability and Measure by Patrick Billingsley, p. 70). While this definition is easier to understand and closer to the literal meaning of "almost surely/with probability $$1$$", it is the definition $$(2)$$ that is more intuitive for comparing the two concepts under consideration.

By definition, for any sequence of events $$\{B_n\}$$, the event $$[B_n \text{ i.o.}]$$ is another way of writing the limit superior of $$\{B_n\}$$, denoted by $$\limsup_n B_n$$, which is defined as $$\cap_{n = 1}^\infty\cup_{k = n}^\infty B_k$$. This definition illustrates the notation "i.o. (infinitely often)": $$\omega$$ lies in $$\limsup_n B_n$$ if and only if $$\omega$$ lies in infinitely many of the $$B_n$$. It can be shown by axioms of probability measure that \begin{align} \limsup_n P(B_n) \leq P\left(\limsup_n B_n\right) = P[B_n \text{ i.o.}]. \tag{3} \end{align}

In view of $$(3)$$, if $$(2)$$ holds, then $$(1)$$ holds automatically, that is, convergence almost surely implies convergence in probability. But the reverse is obviously not always true.

With the above setup, let me comment your interpretations: for convergence in probability, you stated:

The intuition is that for $$n$$ larger and larger there is higher probability that $$\bar{X}_n$$ takes value closer and closer to $$\mu$$. However, it is still possible that for any $$\epsilon > 0$$, we have $$|\bar{X}_n - \mu| > \epsilon$$ occurs an infinite number of times although at infrequent intervals.

Both statements are not precise. $$(1)$$ does not guarantee the monotonicity of how $$\bar{X}_n$$ approaches $$\mu$$, so the first statement is wrong. The second statement is somewhat an extrapolation: $$(1)$$ (i.e., $$P[|\bar{X}_n - \mu| > \epsilon] \to 0$$) does not tell you anything about how many $$|\bar{X}_n - \mu| > \epsilon$$ happened (as mentioned in section 1 above, without knowing specific $$\omega$$, it is impossible to judge if what you stated is right or wrong: $$[|\bar{X}_n - \mu| > \epsilon]$$ may or may not happen for infinitely many times, it is just inconclusive simply based on the convergence condition), the stake here is the probability of $$[|\bar{X}_n - \mu| > \epsilon]$$, which tends to negligible as $$n$$ goes to infinity. So the only conclusion you can make from this is that for all sufficiently large $$n$$, the probability of the event $$[|\bar{X}_n - \mu| > \epsilon]$$ can be made arbitrarily small (but there is no monotonicity).

For convergence almost surely, you stated:

The SLLN claims that as $$n \rightarrow \infty$$, than $$\bar{X}_n \xrightarrow{a.s.} E[X]=\mu$$. However, we can still have $$|\bar{X}_n-\mu|>\epsilon$$, but this will happen a finite number of times.

As before, the inaccuracy here is that you interpreted it without touching the keyword "probability". $$(2)$$ doesn't assert that $$[|\bar{X}_n - \mu| > \epsilon \text{ i.o.}]$$ is an empty set: $$[|\bar{X}_n - \mu| > \epsilon]$$ may still happen for infinitely many times -- it is just the probability of it is $$0$$. Apart from it, what you did get right is that the complement of the event $$[|\bar{X}_n - \mu| > \epsilon \text{ i.o.}]$$ is the limit inferior event $$\liminf_n [|\bar{X}_n - \mu| \leq \epsilon] = \cup_{n = 1}^\infty\cap_{k = n}^\infty[|\bar{X}_k - \mu| \leq \epsilon]$$, which means that $$[|\bar{X}_n - \mu| > \epsilon]$$ happens for only finitely many times. That said, your interpretation on SLLN is correct after appending the qualifier "with probability $$1$$": $$[|\bar{X}_n - \mu| > \epsilon]$$ will happen a finite number of times with probability $$1$$.

This is a supplement to Zhanxiong's comprehensive answer that addressed OP's concerns.

It would be apt to have a brief recollection of the concepts of almost everywhere convergence and convergence in probability.

$$\bullet$$ Consider a measure space $$(X, \boldsymbol{\mathfrak A}, \mu).$$ A sequence of $$\boldsymbol{\mathfrak A}$$-measurable extended real-valued functions $$\langle f_n\rangle_{n\in\mathbb N}$$ on $$D\in \boldsymbol{\mathfrak A}$$ converges almost everywhere to $$f:=\lim_{n\to\infty} f_n(x)$$ if there exists a null set $$N\subset D$$ such that $$f$$ exists for all $$x\in D\setminus N$$ and $$f(x)\in \mathbb R$$ for all $$x\in D\setminus N.$$

$$\bullet$$ $$f_n \overset{\textrm{a.e.}}{\to} f$$ on $$D$$ iff $$\mu\{D:f_n\nrightarrow f \} = 0.$$ Now, by definition, $$\{D:\lim_{n\to\infty} f_n = f\} = \bigcap_{m\in\mathbb N}\bigcup_{N\in\mathbb N}\bigcap_{p\in \mathbb N}\left\{D:|f_{N+p}- f|<\frac1m\right\}$$; so using De Morgan's law \begin{align}\mu\{D:f_n\nrightarrow f \} &= \mu\left\{D\setminus\bigcap_{m\in\mathbb N}\bigcup_{N\in\mathbb N}\bigcap_{p\in \mathbb N}\left\{D:|f_{N+p}- f|<\frac1m\right\}\right\}\\ &= \mu\left\{\bigcup_{m\in\mathbb N}\bigcap_{N\in\mathbb N}\bigcup_{p\in \mathbb N}\left\{D\setminus\left\{D:|f_{N+p}- f|<\frac1m\right\}\right\}\right\}\\&= \mu\left\{\bigcup_{m\in\mathbb N}\bigcap_{N\in\mathbb N}\bigcup_{p\in \mathbb N}\left\{D:|f_{N+p}- f|\geq\frac1m\right\}\right\} .\tag 1\label 1\end{align}

As countable union of null sets is null, from $$\eqref 1$$ dictates that for every $$m,$$ $$\mu\left\{\bigcap_{N\in\mathbb N}\bigcup_{n > N}\left\{D:|f_{n}- f|\geq\frac1m\right\}\right\} = 0.$$ That is, $$f_n \overset{\textrm{a.e.}}{\to} f~\textrm{on}~D\iff \mu\left\{\limsup_{n\to \infty}\left\{D:|f_{n}- f|\geq\frac1m\right\}\right\} = 0~~~\forall m\in \mathbb N.\tag 2\label 2$$

$$\bullet$$ If $$\mu(D)<\infty,$$ by continuity property, $$\eqref 2$$ can be modified to $$f_n \overset{\textrm{a.e.}}{\to} f~\textrm{on}~D\iff \lim_{n\to \infty}\mu\left\{D:|f_{n}- f|\geq\frac1m\right\} = 0~\forall m\in \mathbb N.\tag 3\label 3$$

$$\bullet$$ $$\langle f_n\rangle_{n\in\mathbb N}$$ converges in measure to $$f$$ on $$D$$ if for every $$\varepsilon >0, ~\eta >0,$$ there exists $$N_{\varepsilon,\eta}\in \mathbb N$$ such that for $$n\geq N_{\varepsilon,\eta}$$ $$\mu\{D:|f_{n}- f|\geq \varepsilon\}<\eta,\tag 4$$ which is nothing but the re-statement of $$\lim_{n\to\infty}\mu\{D:|f_{n}- f|\geq \varepsilon\} = 0$$ for every $$\varepsilon >0.$$

$$\bullet$$ From $$\eqref 3,$$ it is clear that for $$\mu(D)<\infty,$$ almost everywhere convergence implies convergence in measure. This is always true for probability measures.

$$\bullet$$ What are these two types of convergences trying to convey?

It is clear almost everywhere/almost surely convergence is related with pointwise convergence. That is for a sequence of random variables $$X_n$$ on $$\Omega,$$ $$\mathbb P\left[\lim_{n\to \infty} X_n(\omega) = X(\omega)\right] = 1.$$

$$\bullet$$ Convergence in probability means $$\lim_{n\to\infty}\mathbb P[|X_n-X|<\varepsilon] = 1.$$ The sequence of probabilities $$\langle\mathbb P[|X_n-X|<\varepsilon] \rangle_{n\in\mathbb N}$$ is of concern here.

Convergence in probability requires that for any $$\varepsilon > 0$$ the sequence of functions must be within an $$\varepsilon$$-band of the function $$X$$ over a set of points from the sample space whose probability increases to one as $$n \to\infty.$$

## References:

$$\rm [I]$$ Real Analysis: Theory of Measure and Integration, J. Yeh, World Scientific, $$2014,$$ sec. $$1\S6.$$

$$\rm[II]$$ Introduction to Statistical Limit Theory, Alan M. Polansky, Taylor and Francis, $$2011,$$ sec. $$3.2-3.3.$$