Why does conditional expectation have this property for independent random variables?

For a reference, please see pp. 53-54 of Boucheron, Lugosi, Massart, Concentration Inequalities: A Nonasymptotic Theory of Independence. Let $f: \mathcal{X}^n \to \mathbb{R}$ be a measurable function ($\mathcal{X}$ is some measurable space), and let $X_1, \dots, X_n$ be independent random variables taking values in $\mathcal{X}$.

Claim: Because of Fubini's theorem, one can write that $$\mathbb{E}[f(X_1, \dots, X_n)| X_1, \dots, X_i] = \int\limits_{\mathcal{X}^{n-i}} f(X_1, \dots, X_i, x_{i+1}, \dots, x_n) d\mu_{i+1}(x_{i+1}) \dots d\mu_n(x_n)$$ where for each $i$, $\mu_i$ denotes the probability distribution of $X_i$. Analogously, $$\mathbb{E}[f(X_1, \dots, X_n)|X_1, \dots, X_{i-1}, X_{i+1}, \dots, X_n] = \int\limits_{\mathcal{X}} f(X_1, \dots, X_{i-1}, x_i, X_{i+1}, \dots, X_n)d\mu_i(x_i)$$

This claim is central to the argument used to show the Efron-Stein inequality in the book mentioned above. Analogous claims were also used extensively in several book excerpts about U-statistics which I was reading a couple of months ago, but by now I have forgotten the specific references.

Anyway, this appears to be an important computational property of conditional expectation, and one that has at least two applications in statistics. It seems plausible to me, but even after figuring out what I need to verify, it still doesn't seem like the integrals involved are well-defined.

Question: Why are the equations below well-defined? In particular, how can the left- and right-hand sides be equal if we integrate over some coordinates twice on the left-hand side but integrate over those same coordinates only once on the right-hand side?

For the first formula:
It suffices to show that, for all $S_1 \in \sigma(X_1), \dots, S_i \in \sigma(X_i)$ : $$\int\limits_{S_1 \times \dots \times S_i \times \mathcal{X} \times \dots \times \mathcal{X} } \left( \int_{\mathcal{X}^{n-i}} f(X_1, \dots, X_i, x_{i+1}, \dots, x_n) d\mu_{i+1}(x_{i+1}) \dots d\mu_n(x_n) \right) d\nu(x_1, \dots, x_n) =$$ $$\int\limits_{S_1 \times \dots \times S_i \times \mathcal{X} \times \dots \times \mathcal{X} } f(X_1, \dots, X_n) d\nu(x_1, \dots, x_n) \,,$$ where $\nu$ is meant to denote the joint probability measure on $\mathcal{X}^n$, which is the product measure of all of the $\mu_j$'s because the $X_j$ are independent.

For the second formula:
It suffices to show for all $S_1 \in \sigma(X_1), \dots, S_{i-1} \in \sigma(X_{i-1}), S_{i+1} \in \sigma(X_{i+1}), \dots, S_n \in \sigma(X_n)$ that:

$$\int\limits_{S_1 \times \dots \times S_{i-1} \times \mathcal{X} \times S_{i+1} \times \dots \times S_n} \left( \int\limits_{\mathcal{X}} f(X_1, \dots, X_{i-1}, x_i, X_{i+1}, \dots, X_n) d\mu_i(x_i) \right) d\nu (x_1, \dots, x_n) =$$ $$\int\limits_{S_1 \times \dots \times S_{i-1} \times \mathcal{X} \times S_{i+1} \times \dots \times S_n } f(X_1, \dots, X_n) d\nu(x_1, \dots, x_n) \,.$$

• @Zhanxiong You might very well be right -- I honestly don't remember what exactly I had in mind anymore and today find the notation I used somewhat difficult to read. Staring at it, what you say seems right, "$f$ should be taking stuff to $\mathbb{R}$" after all. If you want to propose or implement an edit of the question, you have my support. Commented Jan 16, 2023 at 2:59
• I will prove my edited version in my answer below. It turns out that if you used appropriate notations, the proof can be quite short. Commented Jan 16, 2023 at 3:01
• I apologize, your question clearly stated that random variables are taking values in $\mathcal{X}$ -- so the original equality is valid. I will edit my answer (though the essence doesn't change at all). Commented Jan 16, 2023 at 3:40

3 Answers

For $$1 \leq i < n$$, denote random vectors $$(X_1, \ldots, X_i)$$ and $$(X_{i + 1}, \ldots, X_n)$$ by $$X$$ and $$Y$$ respectively. Furthermore, denote product measures $$\mu_1 \times \cdots \times \mu_i$$ (on $$\mathcal{X}^i$$) and $$\mu_{i + 1} \times \cdots \times \mu_n$$ (on $$\mathcal{X}^{n - i}$$) by $$\pi_1$$ and $$\pi_2$$ respectively. Under these notations, your first equality is then:

\begin{align*} E[f(X, Y)|X] = \int_{\mathcal{X}^{n - i}}f(X, y)\pi_2(dy). \tag{1} \end{align*}

As the right hand side of $$(1)$$ is a function of $$X$$, it is $$\sigma(X)$$-measurable. To prove $$(1)$$, it is therefore sufficient to show that for any $$G \in \sigma(X)$$, it holds that

\begin{align} \int_G f(X, Y)dP = \int_G\left[\int_{\mathcal{X}^{n - i}}f(X, y)\pi_2(dy)\right]dP. \tag{2} \end{align}

Since every $$G$$ in $$\sigma(X)$$ has the form $$\{\omega \in \Omega: X(\omega) \in H\} =: X^{-1}(H)$$ for some $$H \in \mathscr{X}^i$$ (Theorem 20.1, Probability and Measure by Patrick Billingsley. Here $$\mathscr{X}^i$$ denotes the $$\sigma$$-field generated by open sets in $$\mathcal{X}^i$$), $$(2)$$ is equivalent to:
\begin{align} \int_{X^{-1}(H)}f(X, Y)dP = \int_{X^{-1}(H)}\left[\int_{\mathcal{X}^{n - i}}f(X, y)\pi_2(dy)\right]dP. \tag{2} \end{align}

By the change of variable theorem (Theorem 16.13, Probability and Measure by Patrick Billingsley. Or refer to this Wikipedia page) and the independence of $$X$$ and $$Y$$ (hence the distribution of $$(X, Y)$$ is the product measure $$\pi_1 \times \pi_2$$), the left hand side of $$(2)$$ is: \begin{align} \int_{X^{-1}(H)}f(X, Y)dP &= \int_{(X, Y)^{-1}(H \times \mathcal{X}^{n - i})}f(X, Y)dP \\ &=\int_{H \times \mathcal{X}^{n - i}} f(x, y) (\pi_1 \times \pi_2)(d(x, y)). \tag{3} \end{align}

Similarly, applying the change of variable theorem to the right hand side of $$(2)$$ yields: \begin{align} \int_{X^{-1}(H)}\left[\int_{\mathcal{X}^{n - i}}f(X, y)\pi_2(dy)\right]dP = \int_H\left[\int_{\mathcal{X}^{n - i}}f(x, y)\pi_2(dy)\right]\pi_1(dx). \tag{4} \end{align}

Now the coincidence of the right hand side of $$(3)$$ and the right hand side of $$(4)$$ is exactly the consequence of the Fubini's theorem. This completes the proof.

The proof to your second equality is completely analogous (with slightly more verbose notations).

• This looks perfect! Thank you so much! To clarify, I'm assuming here Probability and Measure here is referring to Patrick Billingsley's book? (at least I used to be familiar with that one) Commented Jan 16, 2023 at 4:02
• @Chill2Macht Yes, it's Billingsley's classical masterpiece. :) Commented Jan 16, 2023 at 4:04

Attempted answer: Fubini's theorem definitely only applies for the product measure, see here. So this is why we definitely need independence in order to be able to use Fubini's theorem in any way.

If we denote the $\sigma$-algebra assumed to exist on $\mathcal{X}$ by $\mathscr{F}$, then it should follow that the $\sigma$-algebra $$\sigma\left(\underbrace{\mathscr{F} \times \dots \times \mathscr{F}}_{n\ \text{times}}\right)=:\sigma(\mathscr{F}^n) \,,$$ the $\sigma$-algebra generated by the $n$-fold (set-wise) Cartesian product of $\mathscr{F}$ (i.e. by sets of the form $F_1 \times \dots \times F_n$ where for each $j$, $F_j \in \mathscr{F}$), is the implied $\sigma$-algebra on $\mathcal{X}^n$.

Write $\mathscr{N}$ (for null) for the trivial $\sigma$-algebra on $\mathcal{X}$, i.e. $\mathscr{N} = \{ \emptyset, \mathcal{X} \}$.

Originally, in a lot of places below I was using the full $\sigma$-algebra, $\mathscr{F}$, when I should instead have been using $\mathscr{N}$, which lead to much of my confusion. One reason why using $\mathscr{F}$ instead of $\mathscr{N}$ is wrong is the following: $\sigma(X_1), \sigma(X_1, X_2), \dots, \sigma(X_1, \dots, X_n)$ is an increasing filtration, i.e. each $\sigma$-algebra contains "more" sets than the $\sigma$-algebra preceding it in the filtration. This follows by using $\mathscr{N}$ where it is used in the places below, but if I had used $\mathscr{F}$ instead, then it would have been a decreasing filtration, which is completely incorrect. For another reason, we want, e.g., that $\sigma(X_1,X_2)$ is "isomorphic" to a $\sigma$-algebra on $\mathcal{X}^2$, which would only make sense/be possible if the "information" on the remaining coordinates was negligible, i.e. could be completely ignored. That follows if one uses $\mathcal{N}$, but the exact opposite is true if one uses $\mathscr{F}$ - in fact, in that case, one has more "information" about the remaining coordinates than coordinates 1 and 2, not less, and certainly not no information.

Moreover, we have that for each $j$, $\sigma(X_j) \subset \mathscr{F}$. So it makes sense to say that the $\sigma$-algebra $\sigma(X_1, \dots, X_i)$, corresponding to conditioning by $X_1, \dots, X_i$, is equal to $$\sigma\left(\sigma(X_1)\times \dots \times \sigma(X_i) \times \underbrace{\mathscr{N} \times \dots \times \mathscr{N}}_{n-i\ \text{times}}\right) =: \sigma((X_1,\dots, X_i) \times \mathscr{N}^{n-i}) \,,$$ the $\sigma$-algebra generated by sets of the form $$S_1 \times \dots \times S_i \times \underbrace{\mathcal{X} \times \dots \times \mathcal{X}}_{n-i\ \text{times}} \,,$$ where for each $1 \le j \le i$, $S_j \in \sigma(X_j)$ (the event $S_j$ is measurable with respect to the $\sigma$-field generated by $X_j$). (Note that the Cartesian of any product with the empty set is again the empty set, hence why we don't also need to add sets where at least one of the $j$th coordinates, for $j > i$, is the empty set, because all such sets are the empty set, and the empty set is in every $\sigma$-algebra.)

Likewise, the $\sigma$-algebra $\sigma(X_1, \dots, X_{i-1}, X_{i+1}, \dots, X_n)$, corresponding to conditioning on every random variable besides $X_i$, is equal to $$\sigma\left( \sigma(X_1) \times \dots \times \sigma(X_{i-1}) \times \mathscr{N} \times \sigma(X_{i+1}) \times \dots \times \sigma(X_n) \right) =: \sigma( (X_1, \dots, X_{i-1}) \times \mathscr{N} \times (X_{i+1} \times X_n)) \,,$$ the $\sigma$-algebra generated by sets of the form $S_1 \times \dots \times S_{i-1} \times \mathcal{X} \times S_{i+1} \times \dots \times S_n$, where for each $j\not=i$, $S_j$ is measurable with respect to the $\sigma$-algebra induced by $X_j$.

This serves at least two points:

1. To clarify that even without assuming independence of the $X_j$, i.e. without assuming that the joint probability measure on $\mathcal{X}^n$ is equal to the product of the marginal probability measures on each copy of $\mathcal{X}$, we should still have that the $\sigma$-algebra on $\mathcal{X}^n$ is (generated by) the product of the $\sigma$-algebra's on each of the respective copies of $\mathcal{X}$.
2. Showing that the measurability criteria corresponding to our conditional expectation operators can be decomposed into simpler component-wise criteria. What I mean by this is that, to check that a random variable is measurable w.r.t. a $\sigma$-algebra generated by some class of sets, it suffices to show that it is measurable with respect to the generating class. So even if arbitrary sets in the $\sigma$-algebras on $\mathcal{X}^n$ in question can have a very complicated form, the generating sets for these $\sigma$-algebras are all fairly simple, at least inasmuch as they are no more complicated than any of the $\mathscr{F}, \sigma(X_1), \dots, \sigma(X_n)$ component-wise.
3. This reduced class of sets to check works not only for checking measurability, but also for checking the definition of conditional expectation (I think).

I have to look up the precise theorem statement in Dudley's Real Analysis and Probability (assuming such a theorem exists) that shows that checking measurability on the generating sets is sufficient, but I have to imagine that the proof is entirely analogous to the proof that it is sufficient to check continuity by checking the pre-images of bases which generate that topology (e.g. open intervals in the case of the usual topology on $\mathbb{R}$).

Therefore an outline of a proof could be as follows:

For the first formula:
To show that the right-hand side satisfies the definition of conditional expectation, it suffices to show that, for all $S_1 \in \sigma(X_1), \dots, S_i \in \sigma(X_i)$ : $$\int\limits_{S_1 \times \dots \times S_i \times \mathcal{X} \times \dots \times \mathcal{X} } \left( \int_{\mathcal{X}^{n-i}} f(X_1, \dots, X_i, x_{i+1}, \dots, x_n) d\mu_{i+1}(x_{i+1}) \dots d\mu_n(x_n) \right) d\nu(x_1, \dots, x_n) = \int\limits_{S_1 \times \dots \times S_i \times \mathcal{X} \times \dots \times \mathcal{X} } f(X_1, \dots, X_n) d\nu(x_1, \dots, x_n) \,,$$ where $\nu$ is meant to denote the joint probability measure on $\mathcal{X}^n$.

It kind of looks like the integral as written on the left-hand might not be well-defined. I still need to think about this more.

Second formula:
To show that the right-hand side satisfies the definition of conditional expectation, it should be sufficient to show that, for all $S_1 \in \sigma(X_1), \dots, S_{i-1} \in \sigma(X_{i-1}), S_{i+1} \in \sigma(X_{i+1}), \dots, S_n \in \sigma(X_n)$:

$$\int\limits_{S_1 \times \dots \times S_{i-1} \times \mathcal{X} \times S_{i+1} \times \dots \times S_n} \left( \int\limits_{\mathcal{X}} f(X_1, \dots, X_{i-1}, x_i, X_{i+1}, \dots, X_n) d\mu_i(x_i) \right) d\nu (x_1, \dots, x_n) = \int\limits_{S_1 \times \dots \times S_{i-1} \times \mathcal{X} \times S_{i+1} \times \dots \times S_n } f(X_1, \dots, X_n) d\nu(x_1, \dots, x_n) \,.$$

The proof is easy: $$\int f(X,Y)dP_{\Omega /Y}=\int f(x,y)dP_{X\circ Y/Y}=\int f(x,y)dP_{X/Y}$$

• Please explain your (non-standard) notation as it doesn't coincide with that of the OP. Commented Jan 15, 2023 at 20:04
• As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.
– Community Bot
Commented Jan 15, 2023 at 23:58
• I think the concern of others is that ideally answers on this website would be self-contained (on the website) as much as possible, so it's preferred to not have an external link or external reference as the only source defining important parts of the answer (such as the notation and its definitions in this instance). Maybe if it's been used in the same way in other posts on this website, you could include a link to those questions/answers inside of your answer here, to make it more self-contained. Commented Jan 16, 2023 at 0:32
• Admittedly I don't remember this question well anymore, but glancing over it again I think my main concern was being explicit about all of the necessary assumptions / hypotheses that need to be made for the conditional distribution to exist (e.g. such-and-such theorem is valid for all such-and-such spaces, and this is a special case of a such-and-such space because...). This answer seems to assume a priori that the conditional densities exist and are well-defined, as does the link given in the comment (the statement they always exist in Rn is not true, and "Doob 1953" is not sufficient to be Commented Jan 16, 2023 at 0:37
• a clear / unambiguous reference (e.g. even if a bibliography were included in the link specifying the specific article or paper written by Doob in 1953, some pointer to the page number or section number or name of relevant result discussed in that reference would be a lot more helpful). Also in that link, the $\Gamma$'s and $\Omega$'s are undefined, but anyway it seems to say that densities (conditional or otherwise) associate a probability to each event ($\omega$?) in the sigma algebra ($\Omega$?), whereas that's just not true, e.g. densities are not even bounded between 0 and 1 in general. Commented Jan 16, 2023 at 0:42