Skip to main content
added 540 characters in body
Source Link
Matthew Drury
  • 36.3k
  • 4
  • 117
  • 146

Here's half the answer for now.

The equation (2.11) is a consequence of the following little equality. For any two random variables $Z_1$ and $Z_2$, and any function $g$

$$ E_{Z_1, Z_2} (g(Z_1, Z_2)) = E_{Z_2}(E_{Z_1 \mid Z_2}(g(Z_1, Z_2) \mid Z_2)) $$

The notation $E_{Z_1, Z_2}$ is the expectation over the joint distribution. The notation $E_{Z_1 \mid Z_2}$ essentially says "integrate over the conditional distribution of $Z_1$ as if $Z_2$ was fixed".

It's easy to verify this in the case that $Z_1$ and $Z_2$ are discrete random variables by just unwinding the definitions involved

$$ \begin{align} E_{Z_2} & (E_{Z_1 \mid Z_2}(g(Z_1, Z_2) \mid Z_2)) \\ &= E_{Z_2} \left( \sum_{z_1} g(z_1, Z_2) Pr(Z_1 = z_1 \mid Z_2 ) \right) \\ &= \sum_{z_2} \left( \sum_{z_1} g(z_1, z_2) Pr(Z_1 = z_1 \mid Z_2 = z_2 ) \right) Pr(Z_2 = z_2) \\ &= \sum_{z_1, z_2} g(z_1, z_2) Pr(Z_1 = z_1 \mid Z_2 = z_2) Pr(Z_2 = z_2) \\ &= \sum_{z_1, z_2} g(z_1, z_2) Pr(Z_1 = z_1, Z_2 = z_2 ) \\ &= E_{Z_1, Z_2} (g(Z_1, Z_2)) \end{align} $$

The continuous case can either be viewed informally as a limit of this argument, or formally verified once all the measure theoretic do-dads are in place.

To unwind the application, take $Z_1 = Y$, $Z_2 = X$, and $g(x, y) = (y - f(x))^2$. Everything lines up exactly.

The assertion (2.12) asks us to consider minimizing

$$ E_X E_{Y \mid X} (Y - f(X))^2 $$

where we are free to choose $f$ as we wish. Again, focusing on the discrete case, and dropping halfway into the unwinding above, we see that we are minimizing

$$ \sum_{x} \left( \sum_{y} (y - f(x))^2 Pr(Y = y \mid X = x) \right) Pr(X = x) $$

Everything inside the big parenthesis is non-negative, and you can minimize a sum of non-negative quantities by minimizing the summands individually. In context, this means that we can choose $f$ to minimize

$$\sum_{y} (y - f(x))^2 Pr(Y = y \mid X = x)$$

individually for each discrete value of $x$. This is exactly the content of what ESL is claiming, only with fancier notation.

Here's half the answer for now.

The equation (2.11) is a consequence of the following little equality. For any two random variables $Z_1$ and $Z_2$, and any function $g$

$$ E_{Z_1, Z_2} (g(Z_1, Z_2)) = E_{Z_2}(E_{Z_1 \mid Z_2}(g(Z_1, Z_2) \mid Z_2)) $$

The notation $E_{Z_1, Z_2}$ is the expectation over the joint distribution. The notation $E_{Z_1 \mid Z_2}$ essentially says "integrate over the conditional distribution of $Z_1$ as if $Z_2$ was fixed".

It's easy to verify this in the case that $Z_1$ and $Z_2$ are discrete random variables by just unwinding the definitions involved

$$ \begin{align} E_{Z_2} & (E_{Z_1 \mid Z_2}(g(Z_1, Z_2) \mid Z_2)) \\ &= E_{Z_2} \left( \sum_{z_1} g(z_1, Z_2) Pr(Z_1 = z_1 \mid Z_2 ) \right) \\ &= \sum_{z_2} \left( \sum_{z_1} g(z_1, z_2) Pr(Z_1 = z_1 \mid Z_2 = z_2 ) \right) Pr(Z_2 = z_2) \\ &= \sum_{z_1, z_2} g(z_1, z_2) Pr(Z_1 = z_1 \mid Z_2 = z_2) Pr(Z_2 = z_2) \\ &= \sum_{z_1, z_2} g(z_1, z_2) Pr(Z_1 = z_1, Z_2 = z_2 ) \\ &= E_{Z_1, Z_2} (g(Z_1, Z_2)) \end{align} $$

The continuous case can either be viewed informally as a limit of this argument, or formally verified once all the measure theoretic do-dads are in place.

To unwind the application, take $Z_1 = Y$, $Z_2 = X$, and $g(x, y) = (y - f(x))^2$. Everything lines up exactly.

The equation (2.11) is a consequence of the following little equality. For any two random variables $Z_1$ and $Z_2$, and any function $g$

$$ E_{Z_1, Z_2} (g(Z_1, Z_2)) = E_{Z_2}(E_{Z_1 \mid Z_2}(g(Z_1, Z_2) \mid Z_2)) $$

The notation $E_{Z_1, Z_2}$ is the expectation over the joint distribution. The notation $E_{Z_1 \mid Z_2}$ essentially says "integrate over the conditional distribution of $Z_1$ as if $Z_2$ was fixed".

It's easy to verify this in the case that $Z_1$ and $Z_2$ are discrete random variables by just unwinding the definitions involved

$$ \begin{align} E_{Z_2} & (E_{Z_1 \mid Z_2}(g(Z_1, Z_2) \mid Z_2)) \\ &= E_{Z_2} \left( \sum_{z_1} g(z_1, Z_2) Pr(Z_1 = z_1 \mid Z_2 ) \right) \\ &= \sum_{z_2} \left( \sum_{z_1} g(z_1, z_2) Pr(Z_1 = z_1 \mid Z_2 = z_2 ) \right) Pr(Z_2 = z_2) \\ &= \sum_{z_1, z_2} g(z_1, z_2) Pr(Z_1 = z_1 \mid Z_2 = z_2) Pr(Z_2 = z_2) \\ &= \sum_{z_1, z_2} g(z_1, z_2) Pr(Z_1 = z_1, Z_2 = z_2 ) \\ &= E_{Z_1, Z_2} (g(Z_1, Z_2)) \end{align} $$

The continuous case can either be viewed informally as a limit of this argument, or formally verified once all the measure theoretic do-dads are in place.

To unwind the application, take $Z_1 = Y$, $Z_2 = X$, and $g(x, y) = (y - f(x))^2$. Everything lines up exactly.

The assertion (2.12) asks us to consider minimizing

$$ E_X E_{Y \mid X} (Y - f(X))^2 $$

where we are free to choose $f$ as we wish. Again, focusing on the discrete case, and dropping halfway into the unwinding above, we see that we are minimizing

$$ \sum_{x} \left( \sum_{y} (y - f(x))^2 Pr(Y = y \mid X = x) \right) Pr(X = x) $$

Everything inside the big parenthesis is non-negative, and you can minimize a sum of non-negative quantities by minimizing the summands individually. In context, this means that we can choose $f$ to minimize

$$\sum_{y} (y - f(x))^2 Pr(Y = y \mid X = x)$$

individually for each discrete value of $x$. This is exactly the content of what ESL is claiming, only with fancier notation.

Source Link
Matthew Drury
  • 36.3k
  • 4
  • 117
  • 146

Here's half the answer for now.

The equation (2.11) is a consequence of the following little equality. For any two random variables $Z_1$ and $Z_2$, and any function $g$

$$ E_{Z_1, Z_2} (g(Z_1, Z_2)) = E_{Z_2}(E_{Z_1 \mid Z_2}(g(Z_1, Z_2) \mid Z_2)) $$

The notation $E_{Z_1, Z_2}$ is the expectation over the joint distribution. The notation $E_{Z_1 \mid Z_2}$ essentially says "integrate over the conditional distribution of $Z_1$ as if $Z_2$ was fixed".

It's easy to verify this in the case that $Z_1$ and $Z_2$ are discrete random variables by just unwinding the definitions involved

$$ \begin{align} E_{Z_2} & (E_{Z_1 \mid Z_2}(g(Z_1, Z_2) \mid Z_2)) \\ &= E_{Z_2} \left( \sum_{z_1} g(z_1, Z_2) Pr(Z_1 = z_1 \mid Z_2 ) \right) \\ &= \sum_{z_2} \left( \sum_{z_1} g(z_1, z_2) Pr(Z_1 = z_1 \mid Z_2 = z_2 ) \right) Pr(Z_2 = z_2) \\ &= \sum_{z_1, z_2} g(z_1, z_2) Pr(Z_1 = z_1 \mid Z_2 = z_2) Pr(Z_2 = z_2) \\ &= \sum_{z_1, z_2} g(z_1, z_2) Pr(Z_1 = z_1, Z_2 = z_2 ) \\ &= E_{Z_1, Z_2} (g(Z_1, Z_2)) \end{align} $$

The continuous case can either be viewed informally as a limit of this argument, or formally verified once all the measure theoretic do-dads are in place.

To unwind the application, take $Z_1 = Y$, $Z_2 = X$, and $g(x, y) = (y - f(x))^2$. Everything lines up exactly.