Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I am reading the Probability Lifesaver and in the introduction to continuous variables it shows the contradiction when summing infinite number of zeros such as the union of signleton events as,
I can understand the concept that we cannot meaningfully sum uncountable events, or that the union of uncountable events is not necessarily the sum of their probabilities. But thereafter,
Why? Why can it be any number from 0 to 1?
The problem here is that there isn't really any concept of an "uncountable sum" for you to have an intuition about! Summation is initially defined as a binary operation, then extended to finite sums by induction, and then extended to countable sums by taking limits. That is as far as it goes. The closest analogy we have to a "sum" over an uncountable set is the integral. We can make some valid probability statements involving the integral that are similar to the quoted claim, and involve integration as a kind of "uncountable sum". Below I will show what you can and can't validly say.
Using integration as our version of the "uncountable sum", there is a rough analogy that holds here in probability theory that mimics the intuitive properties put forward in the quote section. Suppose we have a continuous random variable $X$ with quantile function $Q_X$ and density function $f_X$. Then the norming property of probability gives:
$$\int \limits_\mathbb{R} f_X(x) \ dx = 1.$$
For any $x \in \mathbb{R}$ we have $\mathbb{P}(X=x) = \int_x^x f_X(x) \ dx = 0$ and yet we can get any quantile value $0 \leqslant p \leqslant 1$ by taking:
$$\int \limits_{-\infty}^{Q_X(p)} f_X(x) \ dx = p.$$
This means that the "uncountable sum" (actually an integral) over a bunch of outcomes with zero probability can give us any probability value we want between zero and one. This occurs because our "sum" is really an integral that is "summing" an infinite number of infinitesimally small values. That is essentially what this kind of "intuitive" description is pointing to.
Unfortunately, it is not possible to extend this idea to get an exact analogy to the quoted section. Even taking the integral as our concept for an "uncountable sum", an exact analogy to the quoted section would be something like the following invalid result:
$$\ \ \int \limits_{-\infty}^{Q_X(p)} \mathbb{P}(X=x) \ dx = p. \quad \quad \quad \text{(Invalid equation)}$$
The reason this does not work is that $\mathbb{P}(X=x) = \int_x^x f_X(r) \ dr \neq f_X(x)$. When we extend from the countable domain to the uncountable domain, and start dealing with "uncountable sums" as integrals, we start using infinitesimals. These infinitesimals are small enough that they give zero probability when we integrate over a single value, but they are actually larger than zero, so when we integrate over a larger (uncountable) set of them, they can give a positive value.
Consequently, we can see that the result really comes down to the fact that, once we start using an "uncountable sum" we also need to start using infinitesimals, which are infinitesimally small non-zero values. If we were to translate the quote into a strictly valid observation on mathematics, it would say that even though infinitesimal values look like zero from one perspective (e.g., integrating them over a single value), an integral of infinitesimal values can be any non-zero value.
An example for which $\text{Prob}(X\in[a,b])$ should be able to take any value between 0 and 1 is a uniform distributed variable. In that case if $X \sim \mathcal{U}(0,1)$ then for $0\leq a\leq b\leq1$ you have $$\text{Prob}(X\in[a,b]) =b-a$$
This is a contradiction with
$$\text{Prob}(X\in[a,b]) =\sum_{\lbrace x \in \mathbb{Q} | 0 \leq x \leq 1 \rbrace} 0 = 0$$
A similar question on maths.stackexchange is the following:
Why is $\infty \cdot 0$ not clearly equal to $0$?
The answer there argues in a similar way, if you would treat multiplication of zero and infinite as if you would treat multiplication of finite numbers, then you could argue not just that $\infty \cdot 0 = 0$ but just as well that $\infty \cdot 0 = \infty$, $\infty \cdot 0 = 1$ or anything else.
The solution to the paradox is in considering whether the terms of the operation are correctly used. What is $0$ and what is $\infty$? Are they correctly used, or are they misinterpreted due to some ambiguities?
Steven Miller writes
"Our argument crucially used that the probability of a sum of disjoint events is a sum of the sum of probabilities of the events. This is true if it is a finite sum, or even a countably infinite sum, but not necessarily true if it's an uncountable sum."
So the problem isn't actually in the infinite sum. The issue with the infinite sum of singleton events is not in the uncountable property. Note: we could have a countable infinite sum of all events $X=x$ when we consider rational numbers $\mathbb{Q}$ (which are countable) as the singleton events.
The argument about the infinite sum is not wrong.
Instead, the error is in the definition of a probability space composed of only singleton events with each a zero probability. The problem is that by introducing an event space of singleton events, each with zero probability, we are creating a measure space that does not have total measure 1.
There is nothing wrong with defining a measure space where each rational number $x\in \mathbb{Q}$ between $0$ and $1$ has some constant measure $m\geq 0$. It is just not a probability measure. This is because for $m=0$ the total measure is zero, and for $m>0$ the total measure is infinite.
It's important to realize that probability, by its definition/construction as a mathematical concept, is only countably additive and not "uncountably additive." Generally in mathematics there is no such concept as uncountably additive (there are a few contexts where uncountable sums have been defined, but they aren't applicable to interpreting probability as an uncountable sum, and in those contexts an "uncountable sum of zeros" is always zero). Really, addition always only involves a finite number of terms and any infinite sum is defined in more complicated ways (e.g. as a limit of finite sums).
It could be problematic that in the shared text the author says first "we cannot assert the probability of an uncountable union of disjoint events is the sum of their probabilities" (which is true) and then says "an uncountable sum of zeros can be any number between 0 and 1." The last statement is a reasonable informal/intuitive way to think about it but an "uncountable sum of zeros" isn't generally a valid mathematical object and definitely isn't valid in this context. Maybe later in the text that is clarified. I would worry that the average reader would see this and just be perplexed, thinking that uncountable sums of zeros are a coherent mathematical thing in other contexts and can be equal to whatever we want (in a mathematically coherent way). The concept of an "uncountable sum of zeros" is simply not defined generally speaking (it is defined in some contexts, but always gives a sum of zero then).
It's important to understand that any time we are talking about "uncountable sums of zeros" in this context of probability, we are talking informally and intuitively and not about formally existing mathematical objects. Generally, it is important to distinguish between informal ideas and formal mathematical objects with the latter being described according to established axioms, definitions, etc.
In the context of probability (and measure in general), if we are to think of adding up uncountably many zeros to get the probability of an event, then that would result in uncountable sums of zeros giving any number between 0 and 1. So that is an informal intuitive reason why uncountable sums of zeros don't make sense here. There is no formal mathematical object, an "uncountable sum," that we are referring to here (i.e. it's just not defined in this context).
Consider the Cantor set $C$ and the set of irrationals $A=[0,1]\cap\mathbb Q^c$. If we are considering $X$ to be a uniformly distributed random variable in $[0,1]$, then $P(X\in C)=0$ and $P(X\in A)=1$ (the Lebesgue measure of these sets is firmly established, and that is identical to their probability under the uniform distribution). So these probabilities are not well-defined as uncountable sums of zeros. Similarly we can create an uncountable set of any probability between 0 and 1 that will similarly be thought of as giving an uncountable sum of zeros equal to its probability. E.g. the fat Cantor set $C_{1/4}$ with middle bits of length $1/4^n$ removed at each stage has measure $1/2$ gives $P(X\in C_{1/4})=1/2$. We can construct fat Cantor sets of any measure between 0 and 1, and each is uncountable.
