Skip to main content
1 of 2
Matthew Drury
  • 36.3k
  • 4
  • 117
  • 146

Here's half the answer for now.

The equation (2.11) is a consequence of the following little equality. For any two random variables $Z_1$ and $Z_2$, and any function $g$

$$ E_{Z_1, Z_2} (g(Z_1, Z_2)) = E_{Z_2}(E_{Z_1 \mid Z_2}(g(Z_1, Z_2) \mid Z_2)) $$

The notation $E_{Z_1, Z_2}$ is the expectation over the joint distribution. The notation $E_{Z_1 \mid Z_2}$ essentially says "integrate over the conditional distribution of $Z_1$ as if $Z_2$ was fixed".

It's easy to verify this in the case that $Z_1$ and $Z_2$ are discrete random variables by just unwinding the definitions involved

$$ \begin{align} E_{Z_2} & (E_{Z_1 \mid Z_2}(g(Z_1, Z_2) \mid Z_2)) \\ &= E_{Z_2} \left( \sum_{z_1} g(z_1, Z_2) Pr(Z_1 = z_1 \mid Z_2 ) \right) \\ &= \sum_{z_2} \left( \sum_{z_1} g(z_1, z_2) Pr(Z_1 = z_1 \mid Z_2 = z_2 ) \right) Pr(Z_2 = z_2) \\ &= \sum_{z_1, z_2} g(z_1, z_2) Pr(Z_1 = z_1 \mid Z_2 = z_2) Pr(Z_2 = z_2) \\ &= \sum_{z_1, z_2} g(z_1, z_2) Pr(Z_1 = z_1, Z_2 = z_2 ) \\ &= E_{Z_1, Z_2} (g(Z_1, Z_2)) \end{align} $$

The continuous case can either be viewed informally as a limit of this argument, or formally verified once all the measure theoretic do-dads are in place.

To unwind the application, take $Z_1 = Y$, $Z_2 = X$, and $g(x, y) = (y - f(x))^2$. Everything lines up exactly.

Matthew Drury
  • 36.3k
  • 4
  • 117
  • 146