Here's half the answer for now.


The equation (2.11) is a consequence of the following little equality.  For any two random variables $Z_1$ and $Z_2$, and any function $g$

$$ E_{Z_1, Z_2} (g(Z_1, Z_2)) = E_{Z_2}(E_{Z_1 \mid Z_2}(g(Z_1, Z_2) \mid Z_2)) $$

The notation $E_{Z_1, Z_2}$ is the expectation over the *joint* distribution.  The notation $E_{Z_1 \mid Z_2}$ essentially says "integrate over the conditional distribution of $Z_1$ as if $Z_2$ was fixed".

It's easy to verify this in the case that $Z_1$ and $Z_2$ are discrete random variables by just unwinding the definitions involved

$$
\begin{align}
E_{Z_2} & (E_{Z_1 \mid Z_2}(g(Z_1, Z_2) \mid Z_2)) \\
    &= E_{Z_2} \left( \sum_{z_1} g(z_1, Z_2) Pr(Z_1 = z_1 \mid Z_2 ) \right) \\
    &= \sum_{z_2} \left( \sum_{z_1} g(z_1, z_2) Pr(Z_1 = z_1 \mid Z_2 = z_2 ) \right) Pr(Z_2 = z_2) \\
    &= \sum_{z_1, z_2} g(z_1, z_2) Pr(Z_1 = z_1 \mid Z_2 = z_2) Pr(Z_2 = z_2) \\
    &= \sum_{z_1, z_2} g(z_1, z_2) Pr(Z_1 = z_1, Z_2 = z_2 ) \\
    &= E_{Z_1, Z_2} (g(Z_1, Z_2))
\end{align}
$$

The continuous case can either be viewed informally as a limit of this argument, or formally verified once all the measure theoretic do-dads are in place.


To unwind the application, take $Z_1 = Y$, $Z_2 = X$, and $g(x, y) = (y - f(x))^2$.  Everything lines up exactly.