Why do we need sigma-algebras to define probability spaces?

Question

We have a random experiment with different outcomes forming the sample space $\Omega,$ on which we look with interest at certain patterns, called events $\mathscr{F}.$ Sigma-algebras (or sigma-fields) are made up of events to which a probability measure $\mathbb{P}$ can be assigned. Certain properties are fulfilled, including the inclusion of the null set $\varnothing$ and the entire sample space, and an algebra that describes unions and intersections with Venn diagrams.

Probability is defined as a function between the $\sigma$-algebra and the interval $[0,1]$. Altogether, the triple $(\Omega, \mathscr{F}, \mathbb{P})$ forms a probability space.

Could someone explain in plain English why the probability edifice would collapse if we didn't have a $\sigma$-algebra? They are just wedged in the middle with that impossibly calligraphic "F". I trust they are necessary; I see that an event is different from an outcome, but what would go awry without a $\sigma$-algebras?

The question is: In what type of probability problems the definition of a probability space including a $\sigma$-algebra becomes a necessity?

---

This online document on the Dartmouth University website provides a plain English accessible explanation. The idea is a spinning pointer rotating counterclockwise on a circle of unit perimeter:

We begin by constructing a spinner, which consists of a circle of unit circumference and a pointer as shown in [the] Figure. We pick a point on the circle and label it $0$, and then label every other point on the circle with the distance, say $x$, from $0$ to that point, measured counterclockwise. The experiment consists of spinning the pointer and recording the label of the point at the tip of the pointer. We let the random variable $X$ denote the value of this outcome. The sample space is clearly the interval $[0,1)$. We would like to construct a probability model in which each outcome is equally likely to occur. If we proceed as we did [...] for experiments with a finite number of possible outcomes, then we must assign the probability $0$ to each outcome, since otherwise, the sum of the probabilities, over all of the possible outcomes, would not equal 1. (In fact, summing an uncountable number of real numbers is a tricky business; in particular, in order for such a sum to have any meaning, at most countably many of the summands can be different than $0$.) However, if all of the assigned probabilities are $0$, then the sum is $0$, not $1$, as it should be.

So if we assigned to each point any probability, and given that there is an (uncountably) infinity number of points, their sum would add up to $> 1$.

Answer 1

To Xi'an's first point: When you're talking about $\sigma$-algebras, you're asking about measurable sets, so unfortunately any answer must focus on measure theory. I'll try to build up to that gently, though.

A theory of probability admitting all subsets of uncountable sets will break mathematics

Consider this example. Suppose you have a unit square in $\mathbb{R}^2$, and you're interested in the probability of randomly selecting a point that is a member of a specific set in the unit square. In lots of circumstances, this can be readily answered based on a comparison of areas of the different sets. For example, we can draw some circles, measure their areas, and then take the probability as the fraction of the square falling in the circle. Very simple.

But what if the area of the set of interest is not well-defined?

If the area is not well-defined, then we can reason to two different but completely valid (in some sense) conclusions about what the area is. So we could have $P(A)=1$ on the one hand and $P(A)=0$ on the other hand, which implies $0=1$. This breaks all of math beyond repair. You can now prove $5<0$ and a number of other preposterous things. Clearly this isn't too useful.

$\boldsymbol{\sigma}$-algebras are the patch that fixes math

What is a $\sigma$-algebra, precisely? It's actually not that frightening. It's just a definition of which sets may be considered as events. Elements not in $\mathscr{F}$ simply have no defined probability measure. Basically, $\sigma$-algebras are the "patch" that lets us avoid some pathological behaviors of mathematics, namely non-measurable sets.

The three requirements of a $\sigma$-field can be considered as consequences of what we would like to do with probability: A $\sigma$-field is a set that has three properties:

1. Closure under countable unions.
2. Closure under countable intersections.
3. Closure under complements.

The countable unions and countable intersections components are direct consequences of the non-measurable set issue. Closure under complements is a consequence of the Kolmogorov axioms: if $P(A)=2/3$, $P(A^c)$ ought to be $1/3$. But without (3), it could happen that $P(A^c)$ is undefined. That would be strange. Closure under complements and the Kolmogorov axioms let us to say things like $P(A\cup A^c)=P(A)+1-P(A)=1$.

Finally, We are considering events in relation to $\Omega$, so we further require that $\Omega\in\mathscr{F}$

Good news: $\boldsymbol{\sigma}$-algebras are only strictly necessary for uncountable sets

But! There's good news here, also. Or, at least, a way to skirt the issue. We only need $\sigma$-algebras if we're working in a set with uncountable cardinality. If we restrict ourselves to countable sets, then we can take $\mathscr{F}=2^\Omega$ the power set of $\Omega$ and we won't have any of these problems because for countable $\Omega$, $2^\Omega$ consists only of measurable sets. (This is alluded to in Xi'an's second comment.) You'll notice that some textbooks will actually commit a subtle sleight-of-hand here, and only consider countable sets when discussing probability spaces.

Additionally, in geometric problems in $\mathbb{R}^n$, it's perfectly sufficient to only consider $\sigma$-algebras composed of sets for which the $\mathcal{L}^n$ measure is defined. To ground this somewhat more firmly, $\mathcal{L}^n$ for $n=1,2,3$ corresponds to the usual notions of length, area and volume. So what I'm saying in the previous example is that the set needs to have a well-defined area for it to have a geometric probability assigned to it. And the reason is this: if we admit non-measureable sets, then we can end up in situations where we can assign probability 1 to some event based on some proof, and probability 0 to the same event event based on some other proof.

But don't let the connection to uncountable sets confuse you! A common misconception that $\sigma$-algebras are countable sets. In fact, they may be countable or uncountable. Consider this illustration: as before, we have a unit square. Define $$\mathscr{F}=\text{All subsets of the unit square with defined $\mathcal{L}^2$ measure}.$$ You can draw a square $B$ with side length $s$ for all $s \in (0,1)$, and with one corner at $(0,0)$. It should be clear that this square is a subset of the unit square. Moreover, all of these squares have defined area, so these squares are elements of $\mathscr{F}$. But it should also be clear that there are uncountably many squares $B$: the number of such squares is uncountable, and each square has defined Lebesgue measure.

So as a practical matter, simply making that observation is often enough to make the observation that you only consider Lebesgue-measurable sets to gain headway against the problem of interest.

But wait, what's a non-measurable set?

I'm afraid I can only shed a little bit of light on this myself. But the Banach-Tarski paradox (sometimes the "sun and pea" paradox) can help us some:

Given a solid ball in 3‑dimensional space, there exists a decomposition of the ball into a finite number of disjoint subsets, which can then be put back together in a different way to yield two identical copies of the original ball. Indeed, the reassembly process involves only moving the pieces around and rotating them, without changing their shape. However, the pieces themselves are not "solids" in the usual sense, but infinite scatterings of points. The reconstruction can work with as few as five pieces.

A stronger form of the theorem implies that given any two "reasonable" solid objects (such as a small ball and a huge ball), either one can be reassembled into the other. This is often stated informally as "a pea can be chopped up and reassembled into the Sun" and called the "pea and the Sun paradox".1

So if you're working with probabilities in $\mathbb{R}^3$ and you're using the geometric probability measure (the ratio of volumes), you want to work out the probability of some event. But you'll struggle to define that probability precisely, because you can rearrange the sets of your space to change volumes! If probability depends on volume, and you can change the volume of the set to be the size of the sun or the size of a pea, then the probability will also change. So no event will have a single probability ascribed to it. Even worse, you can rearrange $S\in\Omega$ such that the volume of $S$ has $V(S)>V(\Omega)$, which implies that the geometric probability measure reports a probability $P(S)>1$, in flagrant violation of the Kolmogorov axioms which require that probability has measure 1.

To resolve this paradox, one could make one of four concessions:

1. The volume of a set might change when it is rotated.
2. The volume of the union of two disjoint sets might be different from the sum of their volumes.
3. The axioms of Zermelo–Fraenkel set theory with the axiom of Choice (ZFC) might have to be altered.
4. Some sets might be tagged "non-measurable", and one would need to check whether a set is "measurable" before talking about its volume.

Option (1) doesn't help use define probabilities, so it's out. Option (2) violates the second Kolmogorov axiom, so it's out. Option (3) seems like a terrible idea because ZFC fixes so many more problems than it creates. But option (4) seems attractive: if we develop a theory of what is and is not measurable, then we will have well-defined probabilities in this problem! This brings us back to measure theory, and our friend the $\sigma$-algebra.

Answer 2

The underlying idea (in very practical terms) is simple. Suppose you are a statistician working with some survey. Lets suppose the survey has some questions about age, but only ask the respondent to identify his age in some given intervals, like $[0,18), [18, 25), [25,34), \dots $. Lets forget the other questions. This questionnaire defines an "event space", your $(\Omega,F)$. The sigma algebra $F$ codifies all information which can be obtained from the questionnaire, so, for the age question (and for now we ignore all other questions), it will contain the interval $[18,25)$ but not other intervals like $[20,30)$, since from the information obtained by the questionnaire we cannot answer question like: do the respondents age belong to $[20,30)$ or not? More generally, a set is an event (belongs to $F$) if and only if we can decide if a sample point belongs to that set or not.

Now, let us define random variables with values in the second event space, $(\Omega', F')$. As an example, take this to be the real line with the usual (Borel) sigma-algebra. Then, an (un-interesting) function which is not a random variable is $f: $"respondents age is a prime number", coding this as 1 if age is prime, 0 else. No, $f^{-1}(1)$ do not belong to $F$, so $f$ is not a random variable. The reason is simple, we cannot decide from the information in the questionnaire if the respondent's age is prime or not! Now you can make more interesting examples yourself.

Why do we require $F$ to be a sigma algebra? Let us say we want to ask two questions of the data, 'is respondent number 3 18 years or older', 'is respondent 3 a female'. Let the questions define two events (sets in $F$) $A$ and $B$, the sets of sample points giving a "yes" answer to that question. Now let us ask the conjunction of the two questions 'is responent 3 a female o18 years or older'. Now that question is represented by the set intersection $A \cap B$. In a similar manner, disjunctions are represented by set union $A \cup B$. Now, requiring closedness for countable intersections and unions lets us ask countable conjunctions or disjunctions. And, negating a question is represented by the complementary set. That gives us a sigma-algebra.

I saw this kind of introduction first in the very good book by Peter Whittle "Probability via expectation" (Springer).

EDIT

Trying to answer whubers question in a comment: "I was a little taken aback at the end, though, when I encountered this asssertion: 'requiring closedness for countable intersections and unions lets us ask countable conjunctions or disjunctions.' This seems to get at the heart of the issue: why would anyone want to construct such an infinitely complicated event? " well, why? Restrict ourselves now to discrete probability, let's say, for convenience, coin tossing. Throwing the coin a finite number of times, all events we can describe using the coin can be expressed via events of the type "head on throw $i$", "tails on throw $i$, and a finite number of "and" or "or". So, in this situation, we do not need $\sigma$-algebras, algebras of sets is enough. So, is there any situation, in this context, where $\sigma$-algebras arise? In practice, even if we can only throw the dice a finite number of times, we develop approximations to probabilities via limit theorems when $n$, the number of throws, grows without bound. So have a look at the proof of the central limit theorem for this case, the Laplace-de Moivre theorem. We can prove via approximations using only algebras, no $\sigma$-algebra should be needed. The weak law of large numbers can be proved via the Chebyshev's inequality, and for that we need only compute variance for finite $n$ cases. But, for the strong law of large numbers, the event we prove has probability one can only be expressed via a countably infinite number of "and" and "or"'s, so for the strong law of large numbers we need $\sigma$-algebras.

But do we really need the strong law of large numbers? According to one answer here, maybe not.

In a way, this points to a very big conceptual difference between the strong and the weak law of large numbers: The strong law is not directly empirically meaningful, since it is about actual convergence, which can never be empirically verified. The weak law, on the other hand, is about the quality of approximation increasing with $n$, with numerical bounds for finite $n$, so is more empirically meaningful.

So, all practical use of discrete probability could do without $\sigma$-algebras. For the continuous case, I am not so sure.

Answer 3

Why do probabilists need $\boldsymbol{\sigma}$-algebra?

The axioms of $\sigma$-algebras are pretty naturally motivated by probability. You want to be able to measure all Venn diagram regions, e.g., $A \cup B$, $(A\cup B)\cap C$. To quote from this memorable answer:

The first axiom is that $\oslash,X\in \sigma$. Well you ALWAYS know the probability of nothing happening ($0$) or something happening ($1$).

The second axiom is closed under complements. Let me offer a stupid example. Again, consider a coin flip, with $X=\{H,T\}$. Pretend I tell you that the $\sigma$ algebra for this flip is $\{\oslash,X,\{H\}\}$. That is, I know the probability of NOTHING happening, of SOMETHING happening, and of a heads but I DON'T know the probability of a tails. You would rightly call me a moron. Because if you know the probability of a heads, you automatically know the probability of a tails! If you know the probability of something happening, you know the probability of it NOT happening (the complement)!

The last axiom is closed under countable unions. Let me give you another stupid example. Consider the roll of a die, or $X=\{1,2,3,4,5,6\}$. What if I were to tell you the $\sigma$ algebra for this is $\{\oslash,X,\{1\},\{2\}\}$. That is, I know the probability of rolling a $1$ or rolling a $2$, but I don't know the probability of rolling a $1$ or a $2$. Again, you would justifiably call me an idiot (I hope the reason is clear). What happens when the sets are not disjoint, and what happens with uncountable unions is a little messier but I hope you can try to think of some examples.

Why do you need countable instead of just finite $\boldsymbol{\sigma}$-additivity though?

Well, it’s not an entirely clean-cut case, but there are some solid reasons why.

Why do probabilists need measures?

At this point, you already have all the axioms for a measure. From $\sigma$-additivity, non-negativity, null empty set, and the domain of $\sigma$-algebra. You might as well require $P$ to be a measure. Measure theory is already justified.

People bring in Vitali’s set and Banach-Tarski to explain why you need measure theory, but I think that’s misleading. Vitali’s set only goes away for (non-trivial) measures that are translation-invariant, which probability spaces do not require. And Banach-Tarski requires rotation-invariance. Analysis people care about them, but probabilists actually don’t.

The raison d’être of measure theory in probability theory is to unify the treatment of discrete and continuous RVs, and moreover, allow for RVs that are mixed and RVs that are simply neither.

Answer 4

I've always understood the whole story like this:

We start off with a space, such as the real line $\mathbb{R}$. We would like to apply our measure to subsets of this space, such as by applying the Lebesgue measure, which measures length. An example would be to measure the length of the subset $[0, 0.5] \cup [0.75, 1]$. For this example the answer is simply $0.5 + 0.25 = 0.75$, which we can obtain fairly easily. We start to wonder whether we can apply the Lebesgue measure to all subsets of the real line.

Unfortunately it doesn't work. There are these pathological sets that simply break down mathematics. If you apply the Lebesgue measure to these sets, you will get inconsistent results. An example of one of these pathological sets, also known as non-measurable sets because they literally can't be measured, are the Vitali Sets.

In order to avoid these crazy sets, we define the measure to only work for a smaller group of subsets, called measurable sets. These are the sets that behave consistently when we apply measures to them. In order to allow us to perform operations with these sets, such as by combining them with unions or taking their complements, we require these measurable sets to form a sigma-algebra amongst themselves. By forming a sigma-algebra, we have formed a sort of safe haven for our measures to operate within, while also allowing us to make reasonable manipulations to get what we want, such as taking unions and complements. This is why we need a sigma-algebra, so that we could draw out a region for the measure to work within, while avoiding non-measurable sets. Notice that if it weren't for these pathological subsets, I can easily define the measure to operate within the power set of the topological space. However, the power set contains all sorts of non-measurable sets, and that's why we have to pick out the measurable ones and make them form a sigma-algebra among themselves.

As you can see, since sigma-algebras are used to avoid non-measurable sets, sets that are finite in size don't actually need a sigma algebra. Let's say you're dealing with a sample space $\Omega = \{1, 2, 3\}$ (this could be all the possible result of a random number generated by a computer). You can see it's pretty much impossible to come up with non-measurable sets with such a sample space. The measure (in this case a probability measure) is well-defined for whatever subset of $\Omega$ you can think of. But we do need to define sigma-algebras for larger sample spaces, such as the real line, so that we can avoid pathological subsets that break down our measures. In order to achieve consistency in the theoretical framework of probability, we require that finite sample spaces also form sigma algebras, where only in which is the probability measure defined. Sigma-algebras in finite sample spaces is a technicality, while sigma-algebras in larger sample spaces such as the real line is a necessity.

One common sigma-algebra we use for the real line is the Borel sigma-algebra. It is formed by all possible open sets, and then taking the complements and unions until the three conditions of a sigma-algebra is achieved. Say if you're constructing the Borel sigma-algebra for $\mathbb{R}[0, 1]$, you do that by listing all possible open sets, such as $(0.5, 0.7), (0.03, 0.05), (0.2, 0.7), ...$ and so on, and as you can imagine there are infinitely many possibilities you can list, and then you take the complements and unions until a sigma-algebra is generated. As you can imagine this sigma algebra is a BEAST. It is unimaginably huge. But the lovely thing about it is that it excludes all the crazy pathological sets that broke down mathematics. Those crazy sets are not in the Borel sigma-algebra. Also, this set is comprehensive enough to include almost every subset that we need. It is hard to think of a subset that is not contained within the Borel sigma-algebra.

And so that is the story of why we need sigma-algebras and Borel sigma-algebras are a common way to implement this idea.

Answer 5

More clarification on the first 2 properties

One quick note on @Sycorax's answer. I'd like to bring more clarity on the term 'uncountable' he/she is using in his/her answer.

Although his/her example on why a σ-algebra can also be 'uncountable' makes a lot of sense, it seems to somehow contradict the first 2 properties of a σ-algebra.
@Sycorax has 'shown' that $\mathscr{F}$ can be a σ-algebra composed of an 'uncountable' number of sets B (squares with an area a ∈ (0,1)).
But wait a minute, if there are an 'uncountable' amount of B squares in $\mathscr{F}$, then $\mathscr{F}$ can potentially have an 'uncountable' amount of unions and intersections, meaning that properties 1. and 2. would be violated, or at least do not apply... So is $\mathscr{F}$ an actual σ-algebra in that case?

That's where the slight clarification of the term 'uncountable' comes into play! In fact, you should read countably infinite instead of 'uncountable'.

Although this might be rather obvious for most of us, I believe it is relevant to highlight the distinction. You can have an infinite number of subsets in your sample space $Ω$, with an infinite number of unions and intersections, but as long as they are countable - even if they are an infinite! (meaning that there exists an injective function from it into the natural numbers*) - and these subsets are closed under unions and intersections (meaning that each union and intersection lands back in $Ω$, so no 'weird' behavior), you're good to go! It is basically what the first two properties mean. You'd of course also need the 3rd property as we're dealing with probabilities, but this was already clearly explained.
And it makes intuitive sense as well! If at some point you've can't count the number of subsets for whatever reason (like a not well-defined area to restate @Sycorax's example) or when the union/intersection of some subsets does not land back in $Ω$, it does imply that you can't measure your sample space, right?

To go a little further, we can write the following $\begin{array}{ccccccccc} Ω & \xrightarrow{\mathbb{P}} & [0,1]\end{array}$ only if $Ω$ is a σ-algebra. Otherwise, $\mathbb{P}$ would not be probability measure - you won't be able to assign a value of 1 to the entire probability space because some interactions (union & intersection) between subsets of $Ω$ would not land back in $Ω$... In fact, $\mathbb{P}$ would not even be a function because you would then be free to assign multiple probability values to the exact same sample (i.e. $\mathbb{P}(x)=y1 =y2$ with $x$ ∈ $Ω$ ; $y1,y2$ ∈ $[0,1]$ ; $y1 \neq y2$), which of course does not make 'any' physical sense and is a great illustration of why you would not be able to correctly measure $Ω$!
Note that $\mathbb{P}$ being an injective function is sufficient here because $\mathbb{P}(x)$ with $x$ ∈ $Ω$ does not have to 'fully cover' $[0,1]$.

*The function mapping the count of unions and intersections of all the subsets of $Ω$ to $\mathbb{N^*}$ is injective because it does not have to 'fully cover' $\mathbb{N^*}$ .