Factoring expected prediction error

Question

I'm trying to understand the derivation of Expected Prediction Error, as described in The Elements of Statistical Learning. Specifically, it says:

$EPE(f) = E(Y - f(X))^2$

By conditioning$^1$ on X, we can write EPE as:

$EPE(f) = E_XE_{Y|X}([Y - f(X)]^2 | X)$

And, a footnote for this says:

$^1$Conditioning here amounts to factoring the joint density $Pr(X,Y) = Pr(Y|X)Pr(X)$ where $Pr(Y|X) = Pr(Y,X)/Pr(X)$ and splitting up the bivariate integral accordingly.

I'm confused on the following:

1. Is $E(Y - f(X))^2$ equivalent to $E((Y - f(X))^2)$ and is $E_XE_{Y|X}(g(X, Y))$ equivalent to $E_X (E_{Y|X}(g(X, Y)))$? I'm not used to seeing $E$ used without an argument surrounded by parentheses.

2. I don't understand why $E_{XY}(g(X,Y)) = E_XE_{Y|X}(g(X,Y) | X)$.

3. What is meant by "splitting up the bivariate integral accordingly" in the footnote?

4. Besides Wikipedia, what are some resources I should be looking at so that I don't get confused by these concepts?

Answer 1

(1) You're pointing out some rather ambiguous notation sometimes used sloppily with expectations.

In my experience, whenever the extra parentheses are suppressed, as in $E(Y-f(X))^2$, this indicates:

$E(Y-f(X))$ is a constant. $E(Y-f(X))^2$ is that constant squared, not the expectation of the random variable $(Y-f(X))^2$. When that latter expectation is desired, $E( (Y-f(X))^2 )$ is written to reduce the ambiguity.

In other cases, when parentheses can be harmlessly suppressed, as in $E_{X}E_{Y|X}(g(X,Y))$, then they are. So, yes,

$$E_{X}E_{Y|X}(g(X,Y))=E_{X}\big(E_{Y|X}(g(X,Y))\big)$$.

(2) This is a special case of the law of total expectation. By the definition of expectation, it can also be seen since it is equivalent to saying

$$ \int_{D} g(x,y)p_{xy}(x,y)dxdy = \int_{D_{x}} p_{x}(x) \left(\int_{D_{y}}g(x,y)p_{y|x}(y)dy\right)dx $$

where $p_{xy}$ is the joint density of $(X,Y)$, $p_{x}$ is the marginal density of $X$, and $p_{y|x}$ is the conditional density of $Y$ given $X=x$. $D_{x},D_{y}$ denote the support of $X,Y$, respectively and $D=D_{x} \times D_{y}$. After noting that

$$p_{xy}(x,y) = p_{y|x}(y)p_{x}(x),$$

the identity becomes pretty clear.

(3) In the integral identity I wrote above, this refers to re-writing the bivariate integral (after you've replaced $p_{xy}(x,y)$ with $p_{y|x}(y)p_{x}(x)$) as separate integrals over $D_{x}$ and $D_{y}$, as on the right hand side.

(4) Review the properties of expectations and conditional/joint densities. Any intermediate text on statistical theory would have this information (e.g. Statistical Inference by Casella and Berger). You may also want to review some of the basic calculus associated with these topics (since expectations are defined as integrals).

Answer 2

There was a similar question at:

https://math.stackexchange.com/questions/513342/expected-squared-prediction-error/623473#623473

I've posted the same answer below for convenience:

$EPE = \int\int {(y-g(x))^2f(x,y)dxdy}$

By Bayes' Theorem $f(x,y)=f(y\,|\,x)\,f(x)$ we have:

$EPE = \int\int {(y-g(x))^2f(y\,|\,x)\,f(x)dxdy}$

Rearranging gives:

$EPE = \int f(x)\;\left(\,\int (y-g(x))^2f(y\,|\,x)dy\,\right)\;dx$

Using definition of $E_x$ we get:

$EPE = E_x(\;\int (y-g(x))^2f(y\,|\,x)dy\;) $

Using definition of $E_{Y\,|\,X}$ we get:

$EPE = E_x(\;\;\;E_{Y\,|\,X}(\,(Y-g(X))^2\,|\,X\,)\;\;\;) $

Or an even shorter notation:

$EPE = E_xE_{Y\,|\,X}(\,[Y-g(X)]^2\,|\,X\,) $