Why does conditional expectation have this property for independent random variables?

Question

For a reference, please see pp. 53-54 of Boucheron, Lugosi, Massart, Concentration Inequalities: A Nonasymptotic Theory of Independence. Let $f: \mathcal{X}^n \to \mathbb{R}$ be a measurable function ($\mathcal{X}$ is some measurable space), and let $X_1, \dots, X_n$ be independent random variables taking values in $\mathcal{X}$.

Claim: Because of Fubini's theorem, one can write that $$\mathbb{E}[f(X_1, \dots, X_n)| X_1, \dots, X_i] = \int\limits_{\mathcal{X}^{n-i}} f(X_1, \dots, X_i, x_{i+1}, \dots, x_n) d\mu_{i+1}(x_{i+1}) \dots d\mu_n(x_n) $$ where for each $i$, $\mu_i$ denotes the probability distribution of $X_i$. Analogously, $$\mathbb{E}[f(X_1, \dots, X_n)|X_1, \dots, X_{i-1}, X_{i+1}, \dots, X_n] = \int\limits_{\mathcal{X}} f(X_1, \dots, X_{i-1}, x_i, X_{i+1}, \dots, X_n)d\mu_i(x_i) $$

This claim is central to the argument used to show the Efron-Stein inequality in the book mentioned above. Analogous claims were also used extensively in several book excerpts about U-statistics which I was reading a couple of months ago, but by now I have forgotten the specific references.

Anyway, this appears to be an important computational property of conditional expectation, and one that has at least two applications in statistics. It seems plausible to me, but even after figuring out what I need to verify, it still doesn't seem like the integrals involved are well-defined.

Question: Why are the equations below well-defined? In particular, how can the left- and right-hand sides be equal if we integrate over some coordinates twice on the left-hand side but integrate over those same coordinates only once on the right-hand side?

For the first formula:
It suffices to show that, for all $S_1 \in \sigma(X_1), \dots, S_i \in \sigma(X_i)$ : $$\int\limits_{S_1 \times \dots \times S_i \times \mathcal{X} \times \dots \times \mathcal{X} } \left( \int_{\mathcal{X}^{n-i}} f(X_1, \dots, X_i, x_{i+1}, \dots, x_n) d\mu_{i+1}(x_{i+1}) \dots d\mu_n(x_n) \right) d\nu(x_1, \dots, x_n) =$$ $$ \int\limits_{S_1 \times \dots \times S_i \times \mathcal{X} \times \dots \times \mathcal{X} } f(X_1, \dots, X_n) d\nu(x_1, \dots, x_n) \,,$$ where $\nu$ is meant to denote the joint probability measure on $\mathcal{X}^n$, which is the product measure of all of the $\mu_j$'s because the $X_j$ are independent.

For the second formula:
It suffices to show for all $S_1 \in \sigma(X_1), \dots, S_{i-1} \in \sigma(X_{i-1}), S_{i+1} \in \sigma(X_{i+1}), \dots, S_n \in \sigma(X_n)$ that:

$$\int\limits_{S_1 \times \dots \times S_{i-1} \times \mathcal{X} \times S_{i+1} \times \dots \times S_n} \left( \int\limits_{\mathcal{X}} f(X_1, \dots, X_{i-1}, x_i, X_{i+1}, \dots, X_n) d\mu_i(x_i) \right) d\nu (x_1, \dots, x_n) =$$ $$ \int\limits_{S_1 \times \dots \times S_{i-1} \times \mathcal{X} \times S_{i+1} \times \dots \times S_n } f(X_1, \dots, X_n) d\nu(x_1, \dots, x_n) \,. $$

Answer 1

For $1 \leq i < n$, denote random vectors $(X_1, \ldots, X_i)$ and $(X_{i + 1}, \ldots, X_n)$ by $X$ and $Y$ respectively. Furthermore, denote product measures $\mu_1 \times \cdots \times \mu_i$ (on $\mathcal{X}^i$) and $\mu_{i + 1} \times \cdots \times \mu_n$ (on $\mathcal{X}^{n - i}$) by $\pi_1$ and $\pi_2$ respectively. Under these notations, your first equality is then:

\begin{align*} E[f(X, Y)|X] = \int_{\mathcal{X}^{n - i}}f(X, y)\pi_2(dy). \tag{1} \end{align*}

As the right hand side of $(1)$ is a function of $X$, it is $\sigma(X)$-measurable. To prove $(1)$, it is therefore sufficient to show that for any $G \in \sigma(X)$, it holds that

\begin{align} \int_G f(X, Y)dP = \int_G\left[\int_{\mathcal{X}^{n - i}}f(X, y)\pi_2(dy)\right]dP. \tag{2} \end{align}

Since every $G$ in $\sigma(X)$ has the form $\{\omega \in \Omega: X(\omega) \in H\} =: X^{-1}(H)$ for some $H \in \mathscr{X}^i$ (Theorem 20.1, Probability and Measure by Patrick Billingsley. Here $\mathscr{X}^i$ denotes the $\sigma$-field generated by open sets in $\mathcal{X}^i$), $(2)$ is equivalent to:
\begin{align} \int_{X^{-1}(H)}f(X, Y)dP = \int_{X^{-1}(H)}\left[\int_{\mathcal{X}^{n - i}}f(X, y)\pi_2(dy)\right]dP. \tag{2} \end{align}

By the change of variable theorem (Theorem 16.13, Probability and Measure by Patrick Billingsley. Or refer to this Wikipedia page) and the independence of $X$ and $Y$ (hence the distribution of $(X, Y)$ is the product measure $\pi_1 \times \pi_2$), the left hand side of $(2)$ is: \begin{align} \int_{X^{-1}(H)}f(X, Y)dP &= \int_{(X, Y)^{-1}(H \times \mathcal{X}^{n - i})}f(X, Y)dP \\ &=\int_{H \times \mathcal{X}^{n - i}} f(x, y) (\pi_1 \times \pi_2)(d(x, y)). \tag{3} \end{align}

Similarly, applying the change of variable theorem to the right hand side of $(2)$ yields: \begin{align} \int_{X^{-1}(H)}\left[\int_{\mathcal{X}^{n - i}}f(X, y)\pi_2(dy)\right]dP = \int_H\left[\int_{\mathcal{X}^{n - i}}f(x, y)\pi_2(dy)\right]\pi_1(dx). \tag{4} \end{align}

Now the coincidence of the right hand side of $(3)$ and the right hand side of $(4)$ is exactly the consequence of the Fubini's theorem. This completes the proof.

The proof to your second equality is completely analogous (with slightly more verbose notations).

Answer 2

Attempted answer: Fubini's theorem definitely only applies for the product measure, see here. So this is why we definitely need independence in order to be able to use Fubini's theorem in any way.

If we denote the $\sigma$-algebra assumed to exist on $\mathcal{X}$ by $\mathscr{F}$, then it should follow that the $\sigma$-algebra $$\sigma\left(\underbrace{\mathscr{F} \times \dots \times \mathscr{F}}_{n\ \text{times}}\right)=:\sigma(\mathscr{F}^n) \,,$$ the $\sigma$-algebra generated by the $n$-fold (set-wise) Cartesian product of $\mathscr{F}$ (i.e. by sets of the form $F_1 \times \dots \times F_n$ where for each $j$, $F_j \in \mathscr{F}$), is the implied $\sigma$-algebra on $\mathcal{X}^n$.

Write $\mathscr{N}$ (for null) for the trivial $\sigma$-algebra on $\mathcal{X}$, i.e. $\mathscr{N} = \{ \emptyset, \mathcal{X} \}$.

Originally, in a lot of places below I was using the full $\sigma$-algebra, $\mathscr{F}$, when I should instead have been using $\mathscr{N}$, which lead to much of my confusion. One reason why using $\mathscr{F}$ instead of $\mathscr{N}$ is wrong is the following: $\sigma(X_1), \sigma(X_1, X_2), \dots, \sigma(X_1, \dots, X_n)$ is an increasing filtration, i.e. each $\sigma$-algebra contains "more" sets than the $\sigma$-algebra preceding it in the filtration. This follows by using $\mathscr{N}$ where it is used in the places below, but if I had used $\mathscr{F}$ instead, then it would have been a decreasing filtration, which is completely incorrect. For another reason, we want, e.g., that $\sigma(X_1,X_2)$ is "isomorphic" to a $\sigma$-algebra on $\mathcal{X}^2$, which would only make sense/be possible if the "information" on the remaining coordinates was negligible, i.e. could be completely ignored. That follows if one uses $\mathcal{N}$, but the exact opposite is true if one uses $\mathscr{F}$ - in fact, in that case, one has more "information" about the remaining coordinates than coordinates 1 and 2, not less, and certainly not no information.

Moreover, we have that for each $j$, $\sigma(X_j) \subset \mathscr{F}$. So it makes sense to say that the $\sigma$-algebra $\sigma(X_1, \dots, X_i)$, corresponding to conditioning by $X_1, \dots, X_i$, is equal to $$\sigma\left(\sigma(X_1)\times \dots \times \sigma(X_i) \times \underbrace{\mathscr{N} \times \dots \times \mathscr{N}}_{n-i\ \text{times}}\right) =: \sigma((X_1,\dots, X_i) \times \mathscr{N}^{n-i}) \,, $$ the $\sigma$-algebra generated by sets of the form $$S_1 \times \dots \times S_i \times \underbrace{\mathcal{X} \times \dots \times \mathcal{X}}_{n-i\ \text{times}} \,,$$ where for each $1 \le j \le i$, $S_j \in \sigma(X_j)$ (the event $S_j$ is measurable with respect to the $\sigma$-field generated by $X_j$). (Note that the Cartesian of any product with the empty set is again the empty set, hence why we don't also need to add sets where at least one of the $j$th coordinates, for $j > i$, is the empty set, because all such sets are the empty set, and the empty set is in every $\sigma$-algebra.)

Likewise, the $\sigma$-algebra $\sigma(X_1, \dots, X_{i-1}, X_{i+1}, \dots, X_n)$, corresponding to conditioning on every random variable besides $X_i$, is equal to $$\sigma\left( \sigma(X_1) \times \dots \times \sigma(X_{i-1}) \times \mathscr{N} \times \sigma(X_{i+1}) \times \dots \times \sigma(X_n) \right) =: \sigma( (X_1, \dots, X_{i-1}) \times \mathscr{N} \times (X_{i+1} \times X_n)) \,, $$ the $\sigma$-algebra generated by sets of the form $S_1 \times \dots \times S_{i-1} \times \mathcal{X} \times S_{i+1} \times \dots \times S_n$, where for each $j\not=i$, $S_j$ is measurable with respect to the $\sigma$-algebra induced by $X_j$.

This serves at least two points:

1. To clarify that even without assuming independence of the $X_j$, i.e. without assuming that the joint probability measure on $\mathcal{X}^n$ is equal to the product of the marginal probability measures on each copy of $\mathcal{X}$, we should still have that the $\sigma$-algebra on $\mathcal{X}^n$ is (generated by) the product of the $\sigma$-algebra's on each of the respective copies of $\mathcal{X}$.
2. Showing that the measurability criteria corresponding to our conditional expectation operators can be decomposed into simpler component-wise criteria. What I mean by this is that, to check that a random variable is measurable w.r.t. a $\sigma$-algebra generated by some class of sets, it suffices to show that it is measurable with respect to the generating class. So even if arbitrary sets in the $\sigma$-algebras on $\mathcal{X}^n$ in question can have a very complicated form, the generating sets for these $\sigma$-algebras are all fairly simple, at least inasmuch as they are no more complicated than any of the $\mathscr{F}, \sigma(X_1), \dots, \sigma(X_n)$ component-wise.
3. This reduced class of sets to check works not only for checking measurability, but also for checking the definition of conditional expectation (I think).

I have to look up the precise theorem statement in Dudley's Real Analysis and Probability (assuming such a theorem exists) that shows that checking measurability on the generating sets is sufficient, but I have to imagine that the proof is entirely analogous to the proof that it is sufficient to check continuity by checking the pre-images of bases which generate that topology (e.g. open intervals in the case of the usual topology on $\mathbb{R}$).

Therefore an outline of a proof could be as follows:

For the first formula:
To show that the right-hand side satisfies the definition of conditional expectation, it suffices to show that, for all $S_1 \in \sigma(X_1), \dots, S_i \in \sigma(X_i)$ : $$\int\limits_{S_1 \times \dots \times S_i \times \mathcal{X} \times \dots \times \mathcal{X} } \left( \int_{\mathcal{X}^{n-i}} f(X_1, \dots, X_i, x_{i+1}, \dots, x_n) d\mu_{i+1}(x_{i+1}) \dots d\mu_n(x_n) \right) d\nu(x_1, \dots, x_n) = \int\limits_{S_1 \times \dots \times S_i \times \mathcal{X} \times \dots \times \mathcal{X} } f(X_1, \dots, X_n) d\nu(x_1, \dots, x_n) \,,$$ where $\nu$ is meant to denote the joint probability measure on $\mathcal{X}^n$.

It kind of looks like the integral as written on the left-hand might not be well-defined. I still need to think about this more.

Second formula:
To show that the right-hand side satisfies the definition of conditional expectation, it should be sufficient to show that, for all $S_1 \in \sigma(X_1), \dots, S_{i-1} \in \sigma(X_{i-1}), S_{i+1} \in \sigma(X_{i+1}), \dots, S_n \in \sigma(X_n)$:

$$\int\limits_{S_1 \times \dots \times S_{i-1} \times \mathcal{X} \times S_{i+1} \times \dots \times S_n} \left( \int\limits_{\mathcal{X}} f(X_1, \dots, X_{i-1}, x_i, X_{i+1}, \dots, X_n) d\mu_i(x_i) \right) d\nu (x_1, \dots, x_n) = \int\limits_{S_1 \times \dots \times S_{i-1} \times \mathcal{X} \times S_{i+1} \times \dots \times S_n } f(X_1, \dots, X_n) d\nu(x_1, \dots, x_n) \,. $$