What is the difference between something being "true" and 'true with probability 1"?

Question

In the beginning of chapter 2 of Information Theory, Inference and Algorithms, the author says that he will refrain from being unnecessarily rigorous and provides the example of saying that something is "true" instead of saying more formally that said thing is "true with probability 1". Is there a difference between something being "true" and 'true with probability 1" ?

Answer 1

If something is true, then it is true with probability of one. Or at least let us assume that without raising further complications. But something with probability of one is not necessarily true. This is the notion of something being almost surely true.

Example

Suppose we sample uniformly on the interval $x \in [0,1] \subset \mathbb{R}$. Since $P(x=\frac{1}{2}) = 0$, we can infer that $P(\lnot [x = \frac{1}{2}]) = 1 - P(x = \frac{1}{2}) = 1$. It is almost surely the case that you will not sample $x=\frac{1}{2}$, but it isn't impossible.

Example: Dart Throwing

This example is from Wikipedia.

Imagine throwing a dart at a unit square (a square with an area of 1) so that the dart always hits an exact point in the square, in such a way that each point in the square is equally likely to be hit. Since the square has area 1, the probability that the dart will hit any particular subregion of the square is equal to the area of that subregion. For example, the probability that the dart will hit the right half of the square is 0.5, since the right half has area 0.5.

Next, consider the event that the dart hits exactly a point in the diagonals of the unit square. Since the area of the diagonals of the square is 0, the probability that the dart will land exactly on a diagonal is 0. That is, the dart will almost never land on a diagonal (equivalently, it will almost surely not land on a diagonal), even though the set of points on the diagonals is not empty, and a point on a diagonal is no less possible than any other point.

Answer 2

This difference occurs because of the difference between probability and possibility

This answer is based on a longer paper O'Neill (2014) that looks at the interaction of a binary possibility operator and a probability measure. Have a look at this paper if you'd like a more detailed exposition of the subject covered in this answer. You can also find a related answer here.

The basic idea here is that we can have a space of outcomes $\Omega_*$ that are all possible, but some outcomes in the space may be so improbable that we can remove them from consideration while still having probability one for the set of remaining outcomes. For example, if we have a continuous random variable $X$ with support $\mathscr{X}$ then every value in the support is possible, but each specific value also has zero probability of occurring. Thus, if we take some value $x \in \mathscr{X}$ then we know that $x$ is a possible value, but we also have $\mathbb{P}(X \neq x) = 1$. In this case, it is not necessarily true that $X \neq x$ (since $x$ is a possible value) but it is true with probability one (we say that this occurs "almost surely"). You can easily construct examples of the divergence between possibility and probability by using zero-probability events for continuous random variables. The dart-throwing example cited in the other answer here is one instance of this.

This split occurs because of the difference between probability and possibility, and it is something you need to keep in mind when dealing with continuous random variables or in other similar cases. Below I will give you a basic outline of how possibility is represented in the analysis of events, and how it diverges from probability. There is a sizable literature analysing mathematical representations of possibility (this field is known as possibility theory), which is closely related to modal logic and fuzzy set theory. Overviews of this field can be found in Yager (ed) (1982), Kacprzyk and Orlovski (eds) (1987), Dubois and Prade (1993), Terano, Asai and Sugeno (eds) (1992), Zadeh and Kacprzyk (1992) and Dubois (2006).

---

A Brief Description of Possibility Theory

The field of possibility theory is formalised in a similar manner to the field of probability theory. In both cases we have an overall space of outcomes $\Omega$ and a class of events $\mathscr{E} \subseteq \Omega$ on that space. Possibilty theory works with a set operator that measures events on a scale from zero to one, just as in probability theory. The theory is characterised by a possibility function $\nabla$ that obeys a set of axioms that are close to those of probability theory, except that they display different behaviour when applied to unions of disjoint events. The general theory allows for a possibility measure that takes on values between zero and one, but here I will use an all-or-nothing possibility measure taking on values zero or one. (This is partly for illustrative purposes and partly because this special case conforms best to the intuitive idea of possibility.) The properties characterising a possibility measure $\nabla$ and a probability measure $\mathbb{P}$ are as follows (in the last line we refer to a countable set $\mathscr{E}_1,\mathscr{E}_2,\mathscr{E}_3,...$ of disjoint events):

$$\begin{matrix} \text{Possibility measure} \quad \quad & & & & & \text{Probability measure} \quad \\[6pt] \nabla(\varnothing) = 0 \quad \quad \quad & & & & & \mathbb{P}(\varnothing) = 0 \quad \quad \\[6pt] \nabla(\Omega) = 1 \quad \quad \quad & & & & & \mathbb{P}(\Omega) = 1 \quad \quad \\[6pt] \nabla ( \bigcup_i \mathscr{E}_i ) = \max_i \nabla(\mathscr{E}_i) & & & & & \mathbb{P} ( \bigcup_i \mathscr{E}_i ) = \sum_i \mathbb{P}(\mathscr{E}_i) \\[6pt] \end{matrix}$$

As you can see, both functions give zero measure for the empty event $\varnothing$ and unit measure for the full space $\Omega$. However, when it comes to dealing with countable unions of disjoint sets, the two measures differ --- the probability of a countable union of disjoint events is equal to the sum of the probabilities of those events, whereas the possibility measure of a countable union of disjoint events is equal to the maximum of the possibility measures of those events.

The above measure for possibility might look a bit strange, since we are used to thinking of possibility as a binary. The possibility measure is formulated in a more generalised way (allowing any value between zero and one) in order to accommodate fields like fuzzy logic and modal logic. Nevertheless, for present purposes, we can simplify possibility theory to deal only with events classified using the binary categorisation (zero or one) with $\nabla(\mathscr{E}) = 0$ meaning that $\mathscr{E}$ is impossible and $\nabla(\mathscr{E}) = 1$ meaning that $\mathscr{E}$ is possible. (Similarly, we say that an event is certain if its negation is impossible.) The third property above then reduces to saying that the union of a countable set of disjoint events is possible if and only if at least one of the events is possible, which corresponds to our intuition.

(In fact, if all events have possibility measure zero or one then it is possible to extend the properties for the possibility measure to allow the above maximisation property over arbitrary unions, not just countable unions. This extension is not generally possible for the probability measure, as is well known.)

---

Relating Probability and Possibility

In order to relate probability and possibility measures on the same space (and thereby see how they interact) we can begin by taking some overarching space $\Omega$ and considering the set of all possible outcomes defined by $\Omega_* \equiv \{ \omega \in \Omega | \nabla(\{\omega\}) = 1 \}$. We would then usually begin our analysis with the axiom that the set of all possible outcomes has probability one (i.e., that $\mathbb{P}(\Omega_*)=1$). Starting with this axiom we can then establish that all certain events are almost sure and all events with positive probability are possible. However, the converses are not necessarily true --- an almost sure event is not necessarily certain and a possible event does not necessarily have positive probability.

I will leave it as an exercise to you to see if you can derive the above rules from the axiom above. It is relatively simple to formulate an example using a continuous random variable where you have events that are uncertain but have probability one. You can find more detail on the relationships between possibility and probability (with a general exposition pitched at statistical students and practitioners) in O'Neill (2014).

Answer 3

As Galen indicated, this is the concept of almost surely or almost everywhere.

To provide a more general framework, consider a measure space $(X, \mathfrak A, \mu). $ For a measurable set $A\in \mathfrak A, $ a property $\mathsf Q$ holds almost everywhere on $A$, provided there exists a null set $A_0\subset A$ that is $\mu(A_0) = 0,$ and

$$\mathsf Q(x)~~~ \forall x\in A\setminus A_0.\tag 1$$

---

Reference:

$[\rm I]$ Real Analysis, H. L. Royden, P. M. Fitzpatrick, Pearson Education, $2010, $ section $2.5, $ p. $45.$

Answer 4

A probability is just a function that satisfies a set of axioms, and maps subsets of the sample space to real numbers between $0$ and $1$. A widely used one is Kolmogorov axioms. Therefore, if you construct your own function that satisfies that set of axioms, no matter what it is, and you have a set which is mapped to $1$, then (the outcome represented by) that set is said to be true with probability $1$.

But when we say something is true, it is logically true. For example, if you flip a coin, it is true that you get either a head or a tail. It can be considered in the realm of “probability”, but it also has logical meaning.

Answer 5

Consider the (-1,-1)-(+1,+1) sheet with a point sized hole in the origin.

It is true with probability 1 that a random point within (-1,-1)-(+1,+1) in the sheet. But there is a point that meets that definition that isn't on the sheet. So it is not true that a point within (-1,-1)-(+1,+1) is on the sheet.

There is no difference in the finite realm.