My question is about "The elements of statistical learning" book (yup, the one). Right now I am kinda stuck on second chapter at part, where they derive EPE for linear regression (Somewhat related to Confusion about derivation of regression function , but I have more a detailed investigation here =)). Here it is:

enter image description here

Ok, I do not really get where did they get it from, but it is not so important, because I've found notes for the book from authors (A Solution Manual and Notes for: The Elements of Statistical Learning).

There authors try to explain how this formula was derived. But I do not get that either =)

So here is the logic that they have (Just a part, other is not important right now):

enter image description here

I do not really get a lot of things here. Let me first explain how I see variables here:

$ T(\tau)$ - just all samples, basically iid, and distribution would be likelihood of all samples

$X$ - input RV

$Y$ - output RV

$x_0$ - is selected input (independent of $T$, just some random $x$)

$y_0$ - output calculated based on $x_0$ (RV, because it depends on e - normal error)

  1. What is $E_{{y_0}|{x_0}}$, how does it differ from $E_{Y|X}$? I mean $X$ an is input and $Y$ is an output, which have quite straightforward relation $Y = \beta X + e$. So $E_{Y|X}$ is expectation of output based on input. And $E_{{y_0}|{x_0}}$ seems pretty much the same to me, though I kinda understand, that $y_0$ is some output given that we have chosen $x_0$ (as it seems it is constant here, though I guess it is weird condition on constant). But, maybe, it is more broader thing, don't know.

  2. How this is true: $E_T =E_XE_{Y|X} $

  3. And the one that blew my head off: $E_TU_1 = U_1E_T$. What does it mean? =) I understand expectation for Random Variable, but what is $U_1E_T$, how am I supposed to understand it? Seems like $E_TU_1$ is just constant given that we've chosen $y_0$ and $x_0$, but I can be totally wrong here.

I have many more questions, but for now I will stop here. I have ok background in stats and linear algebra, I understand undergrad level books (without measure theory though). But I really can't grasp how authors use expectations.


1 Answer 1


I am trying to answer the first question Suppose $x_0$ and $y_0$ are both in $R^p$. $T$ is the space of training parameters which are sets of pairs $(x_0,y_0)$ such that $y_0=x_0^T\beta + \epsilon$. These sets define exactly the training data and starting from there an linear estimation of $y$ as a function of $x$ is calculated with the linear regression method. Let $g:R^p \mapsto R^p$ be a function. Then $E_{x_0|y_0}g = \int_A\!g(y)\rho(y|x_0)\,\mathrm{d}y$ where $A= \{y=x_0^T\beta + \epsilon, where \epsilon \in N(0,\sigma) \}$ . Here $\rho$ is the conditional probability density function of $y$ when $x=x_0$. In our case $g(y)=E_T(y - \hat{y})^2$

For the second question $X$ and $Y$ denote ordered sets of $R^p$ (the input $x$), respectively the sets of corresponding $y$.

For the third question $E_T U_1 = U_1 E_T$ when $U_1$ is a constant with respect to the integration variable in the estimation integral.

  • $\begingroup$ Hello, Thanks a lot for your answer, it clears many things for me (though most of them were in the book and i just got confused =)). So about first one. Ok, cool, i think i now get what Ey0|x0 (i think you have typo there, right?) means - we are averaging over all possible outcomes, given we have training sample x0 (basically it will integrate over all "e" values, because we will resolve x0 by passing it as parameter). But i still not sure what is Ey|x means then, can you be more specific here? $\endgroup$
    – Andre
    Commented Jan 21, 2017 at 9:57
  • $\begingroup$ Also for the third one - right, i get it now. But it is such frustrating notation here, because basically if U1 is constant with respect to the integration, then EtU1 = U1 (because i see U1ET as function f(x) = U1ET(x), maybe it is just me). $\endgroup$
    – Andre
    Commented Jan 21, 2017 at 9:57
  • $\begingroup$ Do you mean $E_{Y|X}$ ? $\endgroup$
    – Cristi
    Commented Jan 21, 2017 at 18:02
  • $\begingroup$ Yes, sry, I decided to skip Latex thing =) $\endgroup$
    – Andre
    Commented Jan 21, 2017 at 18:18
  • $\begingroup$ I think the notation is a bit unfortunate. The following formula is mentioned $E_T=E_XE_{Y|X}$. T denotes the space over which the expectation is computed, but then X and Y are the random variables involved in the regression problem. A different convention is used in the left side of the equation as compared with the right side. $\endgroup$
    – Cristi
    Commented Jan 21, 2017 at 21:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.