When does the law of large numbers fail?

Question

The question is simply what is stated in the title: When does the law of large numbers fail? What I mean is, in what cases will the frequency of an event not tend to the theoretical probability?

Answer 1

There are two theorems (of Kolmogorov) and both require that the expected value be finite. The first holds when variables are IID, the second, when sampling is independent and the variance of the $X_n$ satisfies

$$\sum_{n=1}^\infty \frac{V(X_n)}{n^2} < \infty$$

Say that all $X_n$ have expected value 0, but their variance is $n^2$ so that the condition obviously fails. What happens then? You can still compute an estimated mean, but that mean will not tend to 0 as you sample deeper and deeper. It will tend to deviate more and more as you keep sampling.

Let's give an example. Say that $X_n$ is uniform $U(-n2^n , n2^n)$ so that the condition above fails epically.

$$\sum_{n=1}^\infty \frac{V(X_n)}{n^2} = \sum_{n=1}^\infty \frac{n^2 2^{2n+2}}{12}\frac{1}{n^2} = \frac{1}{3} \sum_{n=1}^\infty 4^n = \infty.$$

By noting that

$$\bar{X}_n = \frac{X_n}{n} + \frac{n-1}{n}\bar{X}_{n-1},$$

we see by induction that the computed average $\bar{X}_n$ is always within the interval $(-2^n, 2^n)$. By using the same formula for $n+1$, we also see that there is always a chance greater than $1/8$ that $\bar{X}_{n+1}$ lies outside $(-2^n, 2^n)$. Indeed, $\frac{X_{n+1}}{n+1}$ is uniform $U(-2^{n+1},2^{n+1})$ and lies outside $(-2^n, 2^n)$ with probability $1/4$. On the other hand, $\frac{n}{n+1}\bar{X}_n$ is in $(-2^n, 2^n)$ by induction, and by symmetry it is positive with probability $1/2$. From these observations it follows immediately that $\bar{X}_{n+1}$ is greater than $2^n$ or smaller than $-2^n$, each with a probability larger than $1/16$. Since the probability that $|\bar{X}_{n+1}| > 2^n$ is greater than $1/8$, there cannot be convergence to 0 as $n$ goes to infinity.

Now, to specifically answer your question, consider an event $A$. If I understood well, you ask "in what conditions is the following statement false?"

$$ \lim_{n \rightarrow \infty} \frac{1}{n}\sum_{k = 1}^{n} 1_A(X_k) = P(X \in A), \; [P]\;a.s.$$

where $1_A$ is the indicator function of the event $A$, i.e. $1_A(X_k) = 1$ if $X_k \in A$ and $0$ otherwise and the $X_k$ are identically distributed (and distributed like $X$).

We see that the condition above will hold, because the variance of an indicator function is bounded above by 1/4, which is the maximum variance of a Bernouilli 0-1 variable. Still, what can go wrong is the second assumption of the strong law of large numbers, namely independent sampling. If the random variables $X_k$ are not sampled independently then convergence is not ensured.

For example, if $X_k$ = $X_1$ for all $k$ then the ratio will be either 1 or 0, whatever the value of $n$, so convergence does not occur (unless $A$ has probability 0 or 1 of course). This is a fake and extreme example. I am not aware of practical cases where convergence to the theoretical probability will not occur. Still, the potentiality exists if sampling is not independent.