When does the law of large numbers fail?

Question

The question is simply what is stated in the title: When does the law of large numbers fail? What I mean is, in what cases will the frequency of an event not tend to the theoretical probability?

Answer 1

There are two theorems (of Kolmogorov). The first holds when variables are IID, the second, when the variance of the $X_n$ is such that

$$\sum_{n=1}^\infty \frac{V(X_n)}{n^2} < \infty$$

Say that all $X_n$ have expected value 0, but their variance is $n^2$ so that the condition obviously fails. What happens then? You can still compute an estimated mean, but that mean will not tend to 0 as you sample deeper and deeper. It will tend to deviate more and more as you keep sampling.

Let's give an example. Say that $X_n$ is $U(-n2^n , n2^n)$ so that the condition above fails (epically). At every stage, the computed mean $\bar{X}_n$ is within the interval $(-2^n, 2^n)$. Because $\bar{X}_{n+1} = X_{n+1}/(n+1) + n\bar{X}_n/(n+1)$ we see that there is always a chance equal to 1/4 that $X_{n+1}$ lies outside $(-2^n, 2^n)$, so there cannot be convergence to 0 as $n$ goes to infinity.

Now, to specifically answer your question, consider an event $A$. If I understood well, you ask "in what conditions is the following statement false?"

$$ \lim_{n \rightarrow \infty} \frac{1}{n}\sum_{k = 1}^{n} 1_A(X_k) = P(X \in A), \; [P]\;a.s.$$

where $1_A$ is the indicator function of the event $A$, i.e. $1_A(X_k) = 1$ if $X_k \in A$ and $0$ otherwise and the $X_k$ are identically distributed (and distributed like $X$).

We see that the condition above will hold, because the variance of an indicator function is bounded above by 1/4, which is the maximum variance of a Bernouilli 0-1 variable. Still, what can go wrong is the second assumption of the strong law of large numbers, namely independent sampling. If the random variables $X_k$ are not sampled independently then convergence is not ensured.

For example, if $X_k$ = $X_1$ for all $k$ then the ratio will be either 1 or 0, whatever the value of $n$, so convergence does not occur (unless $A$ has probability 0 or 1 of course). This is a fake and extreme example. I am not aware of practical cases where convergence to the theoretical probability will not occur. Still, the potentiality exists if sampling is not independent.