# When does the law of large numbers fail?

The question is simply what is stated in the title: When does the law of large numbers fail? What I mean is, in what cases will the frequency of an event not tend to the theoretical probability?

-

There are two theorems (of Kolmogorov). The first holds when variables are IID, the second, when the variance of the $X_n$ is such that

$$\sum_{n=1}^\infty \frac{V(X_n)}{n^2} < \infty$$

Say that all $X_n$ have expected value 0, but their variance is $n^2$ so that the condition obviously fails. What happens then? You can still compute an estimated mean, but that mean will not tend to 0 as you sample deeper and deeper. It will tend to deviate more and more as you keep sampling.

Let's give an example. Say that $X_n$ is $U(-n2^n , n2^n)$ so that the condition above fails (epically). At every stage, the computed mean $\bar{X}_n$ is within the interval $(-2^n, 2^n)$. Because $\bar{X}_{n+1} = X_{n+1}/(n+1) + n\bar{X}_n/(n+1)$ we see that there is always a chance equal to 1/4 that $X_{n+1}$ lies outside $(-2^n, 2^n)$, so there cannot be convergence to 0 as $n$ goes to infinity.

Now, to specifically answer your question, consider an event $A$. If I understood well, you ask "in what conditions is the following statement false?"

$$\lim_{n \rightarrow \infty} \frac{1}{n}\sum_{k = 1}^{n} 1_A(X_k) = P(X \in A), \; [P]\;a.s.$$

where $1_A$ is the indicator function of the event $A$, i.e. $1_A(X_k) = 1$ if $X_k \in A$ and $0$ otherwise and the $X_k$ are identically distributed (and distributed like $X$).

We see that the condition above will hold, because the variance of an indicator function is bounded above by 1/4, which is the maximum variance of a Bernouilli 0-1 variable. Still, what can go wrong is the second assumption of the strong law of large numbers, namely independent sampling. If the random variables $X_k$ are not sampled independently then convergence is not ensured.

For example, if $X_k$ = $X_1$ for all $k$ then the ratio will be either 1 or 0, whatever the value of $n$, so convergence does not occur (unless $A$ has probability 0 or 1 of course). This is a fake and extreme example. I am not aware of practical cases where convergence to the theoretical probability will not occur. Still, the potentiality exists if sampling is not independent.

-
One comment. On wikipedia (lnl page) i have read that the non finiteness of variance only decelerate the convergence of the mean value. Is different from what you states? – emanuele Jun 6 '12 at 14:13
Are you two discussing the same law? The question asks about frequencies of events while this reply seems to focus on the sampling distribution of a mean. Although there is a connection, it hasn't yet appeared explicitly here as far as I can tell. – whuber Jun 6 '12 at 14:53
@whuber True. I focused too much on the title of the question. Thanks for pointing. I updated the answer. – gui11aume Jun 6 '12 at 15:46
@gui11aume i don't understand "We see that the condition above will hold, because the variance of an indicator function is bounded above by 1/4.". What does it means? – emanuele Jun 6 '12 at 16:39
It is not clear what $1_A(X_k)$ means or how it relates to $P(A)$. If the $X_k$ are iid (that is, not just independent), then we might interpret the former as $1_{(X_k \in A)}$ and the latter as $P(X_1 \in A)$, but otherwise... – cardinal Jun 6 '12 at 16:42