Intuition for Conditional Expectation of $\sigma$-algebra

Question

Let $(\Omega,\mathscr{F},\mu)$ be a probability space, given a random variable $\xi:\Omega \to \mathbb{R}$ and a $\sigma$-algebra $\mathscr{G}\subseteq \mathscr{F}$ we can construct a new random variable $E[\xi|\mathscr{G}]$, which is the conditional expectation.

---

What exactly is the intuition for thinking about $E[\xi|\mathscr{G}]$? I understand the intuition for the following:

(i) $E[\xi|A]$ where $A$ is an event (with positive probability).

(ii) $E[\xi|\eta]$ where $\eta$ is a discrete random variable.

But I cannot visualize $E[\xi|\mathscr{G}]$. I understand the mathematics of it, and I understand that it is defined in such a way to generalize the simpler cases that we can visualize. But nonetheless I do not find this way of thinking to be useful. It remains a mysterious object to me.

---

For example, let $A$ be an event with $\mu(A)>0$. Form the $\sigma$-algebra $\mathscr{G} = \{ \emptyset, A, A^c, \Omega\}$, the one generated by $A$. Then $E[\xi|\mathscr{G}](\omega)$ would be equal to $\frac{1}{\mu(A)} \int_A \xi$ if $\omega \in A$, and equal to $\frac{1}{\mu(A^c)} \int_{A^c} \xi$ if $\omega \not \in A$. In other words, $E[\xi|\mathscr{G}](\omega) = E[\xi|A]$ if $\omega\in A$, and $E[\xi|\mathscr{G}](\omega) = E[\xi|A^c]$ if $\omega \in A^c$.

The part which is confusing is that $\omega \in \Omega$, so why do we not just write $E[\xi|\mathscr{G}](\omega) = E[\xi|\Omega] = E[\xi]$? Why do we replace $E[\xi|\mathscr{G}]$ by $E[\xi| A\text{ or } A^c]$ depending on whether or not $\omega\in A$, but not permitted to replace $E[\xi|\mathscr{G}]$ by $E[\xi]$?

---

Note. In responding to this question do not explain this by using the rigorous definition of conditional expectation. I understand that. What I want to understand is what the conditional expectation is supposed to be calculating and why we reject one in place of another.

Answer 1

One way to think about conditional expectation is as a projection onto the $\sigma$-algebra $\mathscr{G}$.

(from Wikimedia commons)

This is actually rigorously true when talking about square-integrable random variables; in this case $\mathbb{E}[\xi|\mathscr{G}]$ is actually the orthogonal projection of the random variable $\xi$ onto the subspace of $L^2(\Omega)$ consisting of random variables measurable with respect to $\mathscr{G}$. And in fact this even turns out to be true in some sense for $L^1$ random variables via approximation by $L^2$ random variables.

(See the comments for references.)

If one considers $\sigma-$algebras as representing how much information we have available (an interpretation which is de rigueur in the theory of stochastic processes), then larger $\sigma-$algebras mean more possible events and thus more information about possible outcomes, while smaller $\sigma-$algebras mean fewer possible events and thus less information about possible outcomes.

Therefore, projecting the $\mathscr{F}$-measurable random variable $\xi$ onto the smaller $\sigma-$algebra $\mathscr{G}$ means taking our best guess for the value of $\xi$ given the more limited information available from $\mathscr{G}$.

In other words, given only the information from $\mathscr{G}$, and not the whole of information from $\mathscr{F}$, $\mathbb{E}[\xi|\mathscr{G}]$ is in a rigorous sense our best possible guess for what the random variable $\xi$ is.

---

With regards to your example, I think you might be confusing random variables and their values. A random variable $X$ is a function whose domain is the event space; it is not a number. In other words, $X: \Omega \to \mathbb{R}$, $X \in \{f\ |\ f: \Omega \to \mathbb{R} \}$ whereas for an $\omega \in \Omega$, $X(\omega)\in\mathbb{R}$.

The notation for conditional expectation, in my opinion, is really bad, because it is a random variable itself, i.e. also a function. In contrast, the (regular) expectation of a random variable is a number. The conditional expectation of a random variable is an entirely different quantity from the expectation of the same random variable, i.e., $\mathbb{E}[\xi|\mathscr{G}]$ doesn't even "type-check" with $\mathbb{E}[\xi]$.

In other words, using the symbol $\mathbb{E}$ to denote both regular and conditional expectation is a very big abuse of notation, which leads to much unnecessary confusion.

All of that being said, note that $\mathbb{E}[\xi|\mathscr{G}](\omega)$ is a number (the value of the random variable $\mathbb{E}[\xi|\mathscr{G}]$ evaluated at the value $\omega$), but $\mathbb{E}[\xi|\Omega]$ is a random variable, but it turns out to be a constant random variable (i.e. trivial degenerate), because the $\sigma$-algebra generated by $\Omega$, $\{ \emptyset, \Omega\}$ is trivial/degenerate, and then technically speaking the constant value of this constant random variable, is $\mathbb{E}[\xi]$, where here $\mathbb{E}$ denotes regular expectation and thus a number, not conditional expectation and thus not a random variable.

Also you seem to be confused about what the notation $\mathbb{E}[\xi|A]$ means; technically speaking it is only possible to condition on $\sigma-$algebras, not on individual events, since probability measures are only defined on complete $\sigma-$algebras, not on individual events. Thus, $\mathbb{E}[\xi|A]$ is just (lazy) shorthand for $\mathbb{E}[\xi|\sigma(A)]$, where $\sigma(A)$ stands for the $\sigma-$algebra generated by the event $A$, which is $\{ \emptyset, A, A^c, \Omega\}$. Note that $\sigma(A) = \mathscr{G} = \sigma(A^c)$; in other words, $\mathbb{E}[\xi|A]$, $\mathbb{E}[\xi|\mathscr{G}]$, and $\mathbb{E}[\xi|A^c]$ are all different ways to denote the exact same object.

Finally I just want to add that the intuitive explanation I gave above explains why the constant value of the random variable $\mathbb{E}[\xi|\Omega]=\mathbb{E}[\xi|\sigma(\Omega)]= \mathbb{E}[\xi| \{ \emptyset, \Omega\}]$ is just the number $\mathbb{E}[\xi]$ -- the $\sigma-$algebra $\{ \emptyset, \Omega\}$ represents the least possible amount of information we could have, in fact essentially no information, so under this extreme circumstance the best possible guess we could have for which random variable $\xi$ is is the constant random variable whose constant value is $\mathbb{E}[\xi]$.

Note that all constant random variables are $L^2$ random variables, and they are all measurable with respect to the trivial $\sigma$-algebra $\{\emptyset, \Omega\}$, so indeed we do have that the constant random $\mathbb{E}[\xi]$ is the orthogonal projection of $\xi$ onto the subspace of $L^2(\Omega)$ consisting of random variables measurable with respect to $\{\emptyset, \Omega\}$, as was claimed.

Answer 2

I am going to try to elaborate what William suggested.

Let $\Omega$ be the sample space of tossing a coin twice. Define the ran. var. $\xi$ to be the num. of heads that occur in the experiment. Clearly, $E[\xi] = 1$. One way of thinking of what $1$, as an expec. value, represents is as the best possible estimate for $\xi$. If we had to take a guess for what value $\xi$ would take, we would guess $1$. This is because $E[(\xi - 1)^2] \leq E[(\xi - a)^2]$ for any real number $a$.

Denote by $A = \{ HT, HH \}$ to be the event that the first outcome is a head. Let $\mathscr{G} = \{ \emptyset, A, A^c, \Omega\}$ be the $\sigma$-alg. gen. by $A$. We think of $\mathscr{G}$ as representing what we know after the first toss. After the first toss, either heads occured, or heads did not occur. Hence, we are either in the event $A$ or $A^c$ after the first toss.

If we are in the event $A$, then the best possible estimate for $\xi$ would be $E[\xi|A] = 1.5$, and if we are in the event $A^c$, then the best possible estimate for $\xi$ would be $E[\xi|A^c] = 0.5$.

Now define the ran. var. $\eta(\omega)$ to be either $1.5$ or $0.5$ depending on whether or not $\omega\in A$. This ran. var. $\eta$, is a better approximation than $1 = E[\xi]$ since $E[(\xi - \eta)^2] \leq E[(\xi -1)^2]$.

What $\eta$ is doing is providing the answer to the question: what is the best estimate of $\xi$ after the first toss? Since we do not know the information after the first toss, $\eta$ will depend on $A$. Once the event $\mathscr{G}$ is revealed to us, after the first toss, the value of $\eta$ is determined and provides the best possible estimate for $\xi$.

The problem with using $\xi$ as its own estimate, i.e. $0=E[(\xi - \xi)^2] \leq E[(\xi - \eta)^2]$ is as follows. $\xi$ is not well-defined after the first toss. Say the outcome of the experiment is $\omega$ with first outcome being heads, we are in the event $A$, but what is $\xi(\omega)=?$ We do not know from just the first toss, that value is ambiguous to us, and so $\xi$ is not well-defined. More formally, we say that $\xi$ is not $\mathscr{G}$-measurable i.e. its value is not well-defined after the first toss. Thus, $\eta$ is the best possible estimate of $\xi$ after the first toss.

Perhaps, somebody here can come up with a more sophisticated example using the sample space $[0,1]$, with $\xi (\omega) = \omega$, and $\mathscr{G}$ some non-trivial $\sigma$-algebra.

Answer 3

Although you request not to use the formal definition, I think that the formal definition is probably the best way of explaining it.

Wikipedia - conditional expectation:

Then a conditional expectation of X given $\displaystyle \scriptstyle {\mathcal {H}}$, denoted as $\displaystyle \scriptstyle \operatorname {E} (X\mid {\mathcal {H}})$, is any $\displaystyle \scriptstyle {\mathcal {H}}$-measurable function ( $\displaystyle \scriptstyle \Omega \to \mathbb {R} ^{n}$) which satisfies:

$\displaystyle \int _{H}\operatorname {E} (X\mid {\mathcal {H}})\;dP=\int _{H}X\;dP\qquad {\text{for each}}\quad H\in {\mathcal {H}}$

Firstly, it is a $\displaystyle \scriptstyle {\mathcal {H}}$-measurable function. Secondly it has to match the expectation over every measurable (sub)set in $\displaystyle \scriptstyle {\mathcal {H}}$. So for an event,A, the sigma algebra is $ \{A,A^C,\emptyset, \Omega\}$, so clearly it is set as you specified in your question for $\omega \in A/A^c$. Similarly for any discrete random variable ( and combinations of them), we list out all primitive events and assign the expectation given that primitive event.

Now consider tossing a coin an infinite number of times, where at each toss $i$, you get $1/2^i$, if your coin is tails then your total winnings are $X=\sum _{i=1}^\infty \frac{1}{2^i}c_i$ where $c_i$ = 1 for tails and 0 for heads. Then X is a real random variable on $[0,1]$. After n coin tosses, you know the value of $X$ to precision $1/2^n$, e.g., after 2 coin tosses it is in $[0,1/4]$, $[1/4,1/2]$, $[1/2,3/4]$ or $[3/4,1]$. After every coin toss, your associated sigma algebra is getting finer and finer, and similarly the conditional expectation of $X$ is getting more and more precise.

Hopefully this example of a real valued random variable with a sequence of sigma algebras getting finer and finer (Filtration) gets you away from the purely event based intuition you are used to, and clarifies its purpose.

Answer 4

To help a little bit more, case ii in your question is a special case that a sigma-algebra being conditioned on is generated by a partition. I am sure you will find that case intuitive as well.