13
$\begingroup$

I'm having some issues with what it means for a probability distribution to converge. Consider this example problem: for a fair dice thrown $6x$ times, for an integer $x$, define $y$ as the probability of seeing each outcome $x$ times.

What I previously thought, was that the law of large numbers says that $$\underset{x \rightarrow \infty}{\lim} y = 1 \tag{1}$$ but it seems the situation is more complicated. We can compute $y$ in a closed form: $$y = \dfrac{(6x)!}{6^{6x} (x!)^6} \quad \Rightarrow \quad \underset{x \rightarrow \infty}{\lim} y = 0\tag{2}$$

The problem above lead me to a second example, which some of you may prefer.

Let us define the random variable $X$ with support on $\{0, 1\}$, such that $\mathbb{P}(X = 1) = 1/6$ and $\mathbb{P}(X = 0) = 5/6$. Using the definition $$\bar{X}_n = \frac{1}{6n}\sum_1^{6n} X_i$$ we have that $$\mathbb{P}(\bar{X}_n = 1/6) = {6n \choose n}\left( \dfrac{1}{6} \right)^n\left( \dfrac{5}{6} \right)^{5n}$$ and it follows that $$\underset{n \rightarrow \infty}{\lim}\mathbb{P}(\bar{X}_n = 1/6) = 0 \tag{3}$$

Now, from the law of large numbers, according to the wikipedia page, $$\mathbb{P}( \underset{n \rightarrow \infty}{\lim} \bar{X}_n = 1/6) = 1 \tag{4}$$ which illustrates the cause of the apparent contradiction between Eqs. (1) and (2) - a difference in the order of the limit and probability. It seems to me that both Eqs. (3) and (4) are valid, which seems strange to me. I would expect that these two equations should have the same value.

Is Eq. 3 valid? If so, why should Eq. 4 also be valid?

$\endgroup$
2
  • 1
    $\begingroup$ I believe stats.stackexchange.com/questions/373157 will resolve most of the confusion about convergence in distribution. About the other issues, you can find much explained here by searching for "law of large numbers" or "WLLN". $\endgroup$
    – whuber
    Apr 9 at 20:15
  • 1
    $\begingroup$ A possible intuitive approach is that the distribution of $\bar X_n$ has expectation $\frac16$ and standard deviation $\sqrt{\frac{5}{216\, n}}$ and the latter tends towards $0$ as $n$ increases. But if you look at $S_n=6n \, \bar X_n$ as the number of say sixes in $6n$ throws, you have $\mathbb P(\bar X_n=\frac16)=\mathbb P(S_n=n)$ and $S_n$ has expectation $n$ and standard deviation $\sqrt{\frac{5n}{6}}$ which is increasing without limit as $n$ increases. $\endgroup$
    – Henry
    Apr 10 at 17:58

4 Answers 4

14
$\begingroup$

Look in more detail at what (3) and (4) are saying

(3) says: the probability that the proportion of 1s is exactly 1/6 goes to zero as the sample size increases (with probability 1)

(4) says: the proportion of 1s gets closer and closer to 1/6 as the sample size increases (with probability 1).

With more symbols: (3) says that for any such sequence you can find an $N$ so that the proportion is never exactly 1/6 for any $n>N$

(4) says that for any given small number $\epsilon$ you can find an $N$ so that the proportion is within $\epsilon$ of 1/6 for any $n>N$

These are not at all contradictory.

$\endgroup$
3
  • 1
    $\begingroup$ Upvoted. One way to see how (3) and (4) coexist is to note that, if we call n the number of rolls and k the number of a specific face, then k/n gets closer and closer to 1/6, but at the same time, k gets farther and farther away from n/6 $\endgroup$
    – Stef
    Apr 10 at 14:19
  • $\begingroup$ "(3) says that for any such sequence you can find an $N$ so that the proportion is never *exactly 1/6* for any $n>N$" Could you clarify? A sequence of what? Truly never? $\endgroup$ Apr 12 at 13:08
  • 1
    $\begingroup$ @paperskilltrees: it's not quite correct. For any $\omega$ in the sample space there is an $N_\omega$ such that it never happens for $n>N_\omega$. The probability that it equals 1/6 infinitely often is zero. $\endgroup$ Apr 13 at 4:51
5
$\begingroup$

Both Equation (3) and Equation (4) are valid, and there is no logical inconsistency. This "paradox" is analogous to the case that without any constraints, $\lim_n E[\xi_n]$ is usually not identical to $E[\lim_n \xi_n]$ (but each expression itself can have its own meaningful value).

For a more straightforward example that presents the same "paradoxical" pattern as the example you found, consider $\xi_1, \ldots, \xi_n \text{ i.i.d. } \sim N(0, 1)$, and let $\bar{X}_n = n^{-1}\sum_{i = 1}^n \xi_i$. Clearly we have $\bar{X}_n \sim N(0, 1/n)$, hence $P(\bar{X}_n = 0) = 0$ for all $n$, whence $\lim_{n \to \infty} P(\bar{X}_n = 0) = 0$. On the other hand, it follows by SLLN that $P(\lim_n \bar{X}_n = 0) = 1$.

$\endgroup$
4
$\begingroup$

The zero probability limit is not surprising when you consider the specificity of the event

The comparison you are looking at here is not surprising. After all, the event that $\bar{X}_n = 1/6$ is equivalent to the specific total $\dot{X}_n = n/6$, which is extremely specific. Indeed, it requires a specific value for the total which is just one possible outcome among a growing list of low probability outcomes. As $n$ becomes large, the probability of any specific value for the total becomes small and so $\mathbb{P}(\bar{X}_n = 1/6) \rightarrow 0$. In fact, for any value $x \in [0,1]$ your model gives the probability limit:

$$\mathbb{P}(\bar{X}_n = x) \rightarrow 0.$$

$\endgroup$
1
$\begingroup$

I wanted to write an answer to clarify the meaning of "zero probability".

Suppose you pick a random number in the interval $(0,1)$ in a uniform manner. The probability that you will pick $\tfrac{1}{2}$ exactly is equal to $0$. How can this be? It is possible, yet equal to zero?

To make sense of this suppose you do an experiment $100$ times in a row and see what numbers you get. Say that you get the number $0.5$ exactly, not rounded, but perfectly. You somehow manage to hit that number perfectly. Then the empirical probability is equal to $\frac{1}{100}$.

Now increase the number of random experiments to $1000$. Since this is a zero-probability event it means that we do not expect the success ratio to scale. Even if, by some miracle, you got $0.5$ exactly again, then your empircal probability would be $\frac{2}{1000}$. Which is even smaller.

Now as you increase the number of experiments $n$ to infinity, in the limit you will get that it is exactly equal to $0$. That is why we say it is a zero-probability event. It is possible, yes, however, if we do a large number of experiments, that ratio will go to zero regardless of being possible.

$\endgroup$
11
  • $\begingroup$ No, this is very far away from a mathematical argument. $\endgroup$
    – Plop
    Apr 12 at 6:06
  • $\begingroup$ @Plop No, what? You are not providing any reasons. Make your case. $\endgroup$ Apr 12 at 19:53
  • $\begingroup$ Ok then. Let $\mu$ be a distribution without atoms. Consider a fixed size sample of iid random variables with distribution $\mu$. Then almost surely, $\frac{1}{2}$ is not an atom of the empirical distribution of the sample. That’s it. Probability theory makes statements about laws, not individual outcomes. The empirical distribution of individual outcomes are meaningless. It’s the law of the empirical distribution that makes sense. $\endgroup$
    – Plop
    Apr 12 at 22:07
  • $\begingroup$ @Plop I do not understand what you are saying. Let $\mu$ be the measure on $(0,1)$, let us not even be general. Consider a fixed size sample, okay $n=1$. So the random variable is $X(\omega) = \omega$ for all $\omega\in (0,1)$. And, so what? $\endgroup$ Apr 13 at 3:24
  • $\begingroup$ So, the empirical distribution $\hat{\mu}$ is a measure-valued random variable such that for all $\omega$, $\hat{\mu}(\omega)$ is the Dirac measure at $\omega$. Probability says things about the law of $\hat{\mu}$, not individual realizations of $\hat{\mu}$. And we have, of course, $\mathbb{P}[\hat{\mu}\mbox{ has an atom at } \frac{1}{2}] = 0$. $\endgroup$
    – Plop
    Apr 13 at 10:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.