4
$\begingroup$

I'm trying to understand the derivation of Expected Prediction Error, as described in The Elements of Statistical Learning. Specifically, it says:

$EPE(f) = E(Y - f(X))^2$

By conditioning$^1$ on X, we can write EPE as:

$EPE(f) = E_XE_{Y|X}([Y - f(X)]^2 | X)$

And, a footnote for this says:

$^1$Conditioning here amounts to factoring the joint density $Pr(X,Y) = Pr(Y|X)Pr(X)$ where $Pr(Y|X) = Pr(Y,X)/Pr(X)$ and splitting up the bivariate integral accordingly.

I'm confused on the following:

  1. Is $E(Y - f(X))^2$ equivalent to $E((Y - f(X))^2)$ and is $E_XE_{Y|X}(g(X, Y))$ equivalent to $E_X (E_{Y|X}(g(X, Y)))$? I'm not used to seeing $E$ used without an argument surrounded by parentheses.

  2. I don't understand why $E_{XY}(g(X,Y)) = E_XE_{Y|X}(g(X,Y) | X)$.

  3. What is meant by "splitting up the bivariate integral accordingly" in the footnote?

  4. Besides Wikipedia, what are some resources I should be looking at so that I don't get confused by these concepts?

$\endgroup$

2 Answers 2

5
$\begingroup$

(1) You're pointing out some rather ambiguous notation sometimes used sloppily with expectations.

In my experience, whenever the extra parentheses are suppressed, as in $E(Y-f(X))^2$, this indicates:

$E(Y-f(X))$ is a constant. $E(Y-f(X))^2$ is that constant squared, not the expectation of the random variable $(Y-f(X))^2$. When that latter expectation is desired, $E( (Y-f(X))^2 )$ is written to reduce the ambiguity.

In other cases, when parentheses can be harmlessly suppressed, as in $E_{X}E_{Y|X}(g(X,Y))$, then they are. So, yes,

$$E_{X}E_{Y|X}(g(X,Y))=E_{X}\big(E_{Y|X}(g(X,Y))\big)$$.

(2) This is a special case of the law of total expectation. By the definition of expectation, it can also be seen since it is equivalent to saying

$$ \int_{D} g(x,y)p_{xy}(x,y)dxdy = \int_{D_{x}} p_{x}(x) \left(\int_{D_{y}}g(x,y)p_{y|x}(y)dy\right)dx $$

where $p_{xy}$ is the joint density of $(X,Y)$, $p_{x}$ is the marginal density of $X$, and $p_{y|x}$ is the conditional density of $Y$ given $X=x$. $D_{x},D_{y}$ denote the support of $X,Y$, respectively and $D=D_{x} \times D_{y}$. After noting that

$$p_{xy}(x,y) = p_{y|x}(y)p_{x}(x),$$

the identity becomes pretty clear.

(3) In the integral identity I wrote above, this refers to re-writing the bivariate integral (after you've replaced $p_{xy}(x,y)$ with $p_{y|x}(y)p_{x}(x)$) as separate integrals over $D_{x}$ and $D_{y}$, as on the right hand side.

(4) Review the properties of expectations and conditional/joint densities. Any intermediate text on statistical theory would have this information (e.g. Statistical Inference by Casella and Berger). You may also want to review some of the basic calculus associated with these topics (since expectations are defined as integrals).

$\endgroup$
3
  • $\begingroup$ Thank you so much! I understand now. I also ordered the book you mentioned. $\endgroup$
    – Alec
    Jan 14, 2012 at 17:23
  • $\begingroup$ Given the context, I'm pretty sure $E(Y - f(X))^2$ is supposed to mean $E((Y - f(X))^2)$ in the book. We're trying to minimize the distance $(Y - f(X))^2$ by finding f. The full context is linked from the question. Let me know if I'm wrong. $\endgroup$
    – Alec
    Jan 15, 2012 at 3:12
  • $\begingroup$ You're right. In Eq (2.10), the integral written there is $E((Y-f(X))^2)$. Apparently $EX^2$ can mean both $[E(X)]^2$ and $E(X^2)$ - I've still seen it both ways. $\endgroup$
    – Macro
    Jan 15, 2012 at 3:17
2
$\begingroup$

There was a similar question at:

https://math.stackexchange.com/questions/513342/expected-squared-prediction-error/623473#623473

I've posted the same answer below for convenience:

$EPE = \int\int {(y-g(x))^2f(x,y)dxdy}$

By Bayes' Theorem $f(x,y)=f(y\,|\,x)\,f(x)$ we have:

$EPE = \int\int {(y-g(x))^2f(y\,|\,x)\,f(x)dxdy}$

Rearranging gives:

$EPE = \int f(x)\;\left(\,\int (y-g(x))^2f(y\,|\,x)dy\,\right)\;dx$

Using definition of $E_x$ we get:

$EPE = E_x(\;\int (y-g(x))^2f(y\,|\,x)dy\;) $

Using definition of $E_{Y\,|\,X}$ we get:

$EPE = E_x(\;\;\;E_{Y\,|\,X}(\,(Y-g(X))^2\,|\,X\,)\;\;\;) $

Or an even shorter notation:

$EPE = E_xE_{Y\,|\,X}(\,[Y-g(X)]^2\,|\,X\,) $

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.