31
$\begingroup$

In the beginning of chapter 2 of Information Theory, Inference and Algorithms, the author says that he will refrain from being unnecessarily rigorous and provides the example of saying that something is "true" instead of saying more formally that said thing is "true with probability 1". Is there a difference between something being "true" and 'true with probability 1" ?

$\endgroup$
7
  • 16
    $\begingroup$ Yes, there is. If $\Omega$ denotes the whole space then $A=\Omega$ is a stronger statement than $P(A)=1$. $\endgroup$
    – drhab
    Oct 3, 2022 at 7:53
  • 11
    $\begingroup$ One of my professors always used the opposite statement: "The probabilty for this event is zero, which doesn't mean it's impossible." Another professor came in super happy one day, saying he had watched a football match the evening before, where the coin tossed at the beginning had landed on the edge, an event that is commonly assigned probability 0, but obviously not impossible. $\endgroup$
    – Sabine
    Oct 3, 2022 at 16:56
  • 16
    $\begingroup$ @Peter There are important examples of the distinction. A uniform random value in the interval $[0,1]$ has probability $1$ of being irrational. It is not true that all values in this interval are irrational. Although an infinite sequence of independent flips of a fair coin has probability $1$ that the limiting proportion of heads is $1/2,$ it is not true that all such sequences have limiting proportions of $1/2.$ $\endgroup$
    – whuber
    Oct 3, 2022 at 17:21
  • 2
    $\begingroup$ fantastic mr fox --> stochastic mr fox? :D $\endgroup$
    – BCLC
    Oct 3, 2022 at 23:13
  • 3
    $\begingroup$ @Sabine I'm not quite sure that's a fair example of the issue though. What's going on there is that the 50/50 model of coin tosses isn't faithfully modeling the physical situation. On the other hand, for a uniform distribution on $[0, 1]$, no matter what result you get when you sample from it, there was a prior probability of 0 that you would get that result. $\endgroup$ Oct 4, 2022 at 9:18

5 Answers 5

44
$\begingroup$

If something is true, then it is true with probability of one. Or at least let us assume that without raising further complications. But something with probability of one is not necessarily true. This is the notion of something being almost surely true.

Example

Suppose we sample uniformly on the interval $x \in [0,1] \subset \mathbb{R}$. Since $P(x=\frac{1}{2}) = 0$, we can infer that $P(\lnot [x = \frac{1}{2}]) = 1 - P(x = \frac{1}{2}) = 1$. It is almost surely the case that you will not sample $x=\frac{1}{2}$, but it isn't impossible.

Example: Dart Throwing

This example is from Wikipedia.

Imagine throwing a dart at a unit square (a square with an area of 1) so that the dart always hits an exact point in the square, in such a way that each point in the square is equally likely to be hit. Since the square has area 1, the probability that the dart will hit any particular subregion of the square is equal to the area of that subregion. For example, the probability that the dart will hit the right half of the square is 0.5, since the right half has area 0.5.

Next, consider the event that the dart hits exactly a point in the diagonals of the unit square. Since the area of the diagonals of the square is 0, the probability that the dart will land exactly on a diagonal is 0. That is, the dart will almost never land on a diagonal (equivalently, it will almost surely not land on a diagonal), even though the set of points on the diagonals is not empty, and a point on a diagonal is no less possible than any other point.

$\endgroup$
3
  • $\begingroup$ A fun interpretation of this is that a model where you can "sample" a set of measure zero means you aren't modelling what we think of as probability, but an extension. $\endgroup$
    – Yakk
    Oct 5, 2022 at 13:32
  • $\begingroup$ "If something is true, then it is true with probability of one. Or at least let us assume that without raising further complications." So your answer to the original question is that there is no difference? The author was just padding their word count? :-) $\endgroup$ Oct 7, 2022 at 8:07
  • 2
    $\begingroup$ @DarrenCook There is a distinct of the converse being made. $[X=a] \implies [Pr(X=a) = 1]$ but $[Pr(X=a) = 1] \not\!\!\!\implies [X = a]$. My answer focuses on the latter, which is not equivalent to the former. $\endgroup$
    – Galen
    Oct 7, 2022 at 14:55
27
$\begingroup$

This difference occurs because of the difference between probability and possibility

This answer is based on a longer paper O'Neill (2014) that looks at the interaction of a binary possibility operator and a probability measure. Have a look at this paper if you'd like a more detailed exposition of the subject covered in this answer. You can also find a related answer here.

The basic idea here is that we can have a space of outcomes $\Omega_*$ that are all possible, but some outcomes in the space may be so improbable that we can remove them from consideration while still having probability one for the set of remaining outcomes. For example, if we have a continuous random variable $X$ with support $\mathscr{X}$ then every value in the support is possible, but each specific value also has zero probability of occurring. Thus, if we take some value $x \in \mathscr{X}$ then we know that $x$ is a possible value, but we also have $\mathbb{P}(X \neq x) = 1$. In this case, it is not necessarily true that $X \neq x$ (since $x$ is a possible value) but it is true with probability one (we say that this occurs "almost surely"). You can easily construct examples of the divergence between possibility and probability by using zero-probability events for continuous random variables. The dart-throwing example cited in the other answer here is one instance of this.

This split occurs because of the difference between probability and possibility, and it is something you need to keep in mind when dealing with continuous random variables or in other similar cases. Below I will give you a basic outline of how possibility is represented in the analysis of events, and how it diverges from probability. There is a sizable literature analysing mathematical representations of possibility (this field is known as possibility theory), which is closely related to modal logic and fuzzy set theory. Overviews of this field can be found in Yager (ed) (1982), Kacprzyk and Orlovski (eds) (1987), Dubois and Prade (1993), Terano, Asai and Sugeno (eds) (1992), Zadeh and Kacprzyk (1992) and Dubois (2006).


A Brief Description of Possibility Theory

The field of possibility theory is formalised in a similar manner to the field of probability theory. In both cases we have an overall space of outcomes $\Omega$ and a class of events $\mathscr{E} \subseteq \Omega$ on that space. Possibilty theory works with a set operator that measures events on a scale from zero to one, just as in probability theory. The theory is characterised by a possibility function $\nabla$ that obeys a set of axioms that are close to those of probability theory, except that they display different behaviour when applied to unions of disjoint events. The general theory allows for a possibility measure that takes on values between zero and one, but here I will use an all-or-nothing possibility measure taking on values zero or one. (This is partly for illustrative purposes and partly because this special case conforms best to the intuitive idea of possibility.) The properties characterising a possibility measure $\nabla$ and a probability measure $\mathbb{P}$ are as follows (in the last line we refer to a countable set $\mathscr{E}_1,\mathscr{E}_2,\mathscr{E}_3,...$ of disjoint events):

$$\begin{matrix} \text{Possibility measure} \quad \quad & & & & & \text{Probability measure} \quad \\[6pt] \nabla(\varnothing) = 0 \quad \quad \quad & & & & & \mathbb{P}(\varnothing) = 0 \quad \quad \\[6pt] \nabla(\Omega) = 1 \quad \quad \quad & & & & & \mathbb{P}(\Omega) = 1 \quad \quad \\[6pt] \nabla ( \bigcup_i \mathscr{E}_i ) = \max_i \nabla(\mathscr{E}_i) & & & & & \mathbb{P} ( \bigcup_i \mathscr{E}_i ) = \sum_i \mathbb{P}(\mathscr{E}_i) \\[6pt] \end{matrix}$$

As you can see, both functions give zero measure for the empty event $\varnothing$ and unit measure for the full space $\Omega$. However, when it comes to dealing with countable unions of disjoint sets, the two measures differ --- the probability of a countable union of disjoint events is equal to the sum of the probabilities of those events, whereas the possibility measure of a countable union of disjoint events is equal to the maximum of the possibility measures of those events.

The above measure for possibility might look a bit strange, since we are used to thinking of possibility as a binary. The possibility measure is formulated in a more generalised way (allowing any value between zero and one) in order to accommodate fields like fuzzy logic and modal logic. Nevertheless, for present purposes, we can simplify possibility theory to deal only with events classified using the binary categorisation (zero or one) with $\nabla(\mathscr{E}) = 0$ meaning that $\mathscr{E}$ is impossible and $\nabla(\mathscr{E}) = 1$ meaning that $\mathscr{E}$ is possible. (Similarly, we say that an event is certain if its negation is impossible.) The third property above then reduces to saying that the union of a countable set of disjoint events is possible if and only if at least one of the events is possible, which corresponds to our intuition.

(In fact, if all events have possibility measure zero or one then it is possible to extend the properties for the possibility measure to allow the above maximisation property over arbitrary unions, not just countable unions. This extension is not generally possible for the probability measure, as is well known.)


Relating Probability and Possibility

In order to relate probability and possibility measures on the same space (and thereby see how they interact) we can begin by taking some overarching space $\Omega$ and considering the set of all possible outcomes defined by $\Omega_* \equiv \{ \omega \in \Omega | \nabla(\{\omega\}) = 1 \}$. We would then usually begin our analysis with the axiom that the set of all possible outcomes has probability one (i.e., that $\mathbb{P}(\Omega_*)=1$). Starting with this axiom we can then establish that all certain events are almost sure and all events with positive probability are possible. However, the converses are not necessarily true --- an almost sure event is not necessarily certain and a possible event does not necessarily have positive probability.

I will leave it as an exercise to you to see if you can derive the above rules from the axiom above. It is relatively simple to formulate an example using a continuous random variable where you have events that are uncertain but have probability one. You can find more detail on the relationships between possibility and probability (with a general exposition pitched at statistical students and practitioners) in O'Neill (2014).

$\endgroup$
10
  • 1
    $\begingroup$ I was not aware of possibility theory. Why isn't it discussed in our uni courses? Thanks for the references. +1. $\endgroup$ Oct 3, 2022 at 11:57
  • 1
    $\begingroup$ It is discussed in university courses on fuzzy logic, modal logic, etc., but these are outside the scope of a standard statistical education. $\endgroup$
    – Ben
    Oct 3, 2022 at 11:58
  • 2
    $\begingroup$ Hmm. We do have special papers on fuzzy logic and Machine Learning but these are not covered in stats curriculum. Anyway, that was a brilliant take to a point where I wasn't even aware it existed. $\endgroup$ Oct 3, 2022 at 12:02
  • 1
    $\begingroup$ +1 for the references and exposition. I'll confess that I am skeptical of the notion of non-binary possibility, however popular it is in fuzzy logic. It must be time for me to do some reading into the foundations of fuzzy logic to confirm or alleviate my ontological concerns. $\endgroup$
    – Galen
    Oct 3, 2022 at 16:27
  • 2
    $\begingroup$ If I understand the above correctly, one could define a classical possibility measure by including a constraint that a possibility measure $\nabla$ maps to $\{ 0, 1 \}$ rather than $[0,1]$. $\endgroup$
    – Galen
    Oct 3, 2022 at 16:48
15
$\begingroup$

As Galen indicated, this is the concept of almost surely or almost everywhere.

To provide a more general framework, consider a measure space $(X, \mathfrak A, \mu). $ For a measurable set $A\in \mathfrak A, $ a property $\mathsf Q$ holds almost everywhere on $A$, provided there exists a null set $A_0\subset A$ that is $\mu(A_0) = 0,$ and

$$\mathsf Q(x)~~~ \forall x\in A\setminus A_0.\tag 1$$


Reference:

$[\rm I]$ Real Analysis, H. L. Royden, P. M. Fitzpatrick, Pearson Education, $2010, $ section $2.5, $ p. $45.$

$\endgroup$
3
  • 2
    $\begingroup$ +1 Yes, this is the formal treatment of the notion. It might be better to read Royden first, but a nice follow up is Measure Theory and Probability Theory, Krishna Athreya and Soumendra Lahiri, 2006. $\endgroup$
    – Galen
    Oct 3, 2022 at 3:04
  • 1
    $\begingroup$ While Athreya, Lahiri was mentioned in references in our semester curriculum, unfortunately I never had the chance to go through that. I have my staple source in Royden; Yeh and occasionally others. In any case, I thought as you already provided an intuitive account of the concept, a formal definition would complete the post here. $\endgroup$ Oct 3, 2022 at 3:14
  • 1
    $\begingroup$ In fact, Royden introduced it using Lebesgue measure. But, evidently, it works in the general framework. $\endgroup$ Oct 3, 2022 at 3:17
7
$\begingroup$

A probability is just a function that satisfies a set of axioms, and maps subsets of the sample space to real numbers between $0$ and $1$. A widely used one is Kolmogorov axioms. Therefore, if you construct your own function that satisfies that set of axioms, no matter what it is, and you have a set which is mapped to $1$, then (the outcome represented by) that set is said to be true with probability $1$.

But when we say something is true, it is logically true. For example, if you flip a coin, it is true that you get either a head or a tail. It can be considered in the realm of “probability”, but it also has logical meaning.

$\endgroup$
2
$\begingroup$

Consider the (-1,-1)-(+1,+1) sheet with a point sized hole in the origin.

It is true with probability 1 that a random point within (-1,-1)-(+1,+1) in the sheet. But there is a point that meets that definition that isn't on the sheet. So it is not true that a point within (-1,-1)-(+1,+1) is on the sheet.

There is no difference in the finite realm.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.