I am reading section for 7.4 in elements of statistical learning, and I have some questions about some definitions and results in this section, and I would like to check if my understanding is correct.

  1. In the definition for generalization error $Err_T=\mathbb{E}_{X^0,Y^0}[L(Y^0,\hat{f}(X^0)|T]$, why do we condition on training set $T$ if $X^0,Y^0$ are simply drawn from the true joint distribution? Is it just because the fit model $\hat{f}$ is dependent on the training set $T$?

  2. They define in-sample error as $Err_{in}=\frac{1}{N}\sum_{i=1}^N E_{Y^0}[L(Y_i^0,\hat{f}(x_i))|T]$. Here, I assume $Y_i^0~p_{Y|X}(\cdot|x_i)$ What is the difference between $\mathbb{E}_\mathbf{y}[Err_{in}]$ and $Err_{in}$, as the latter already has an expectation over $Y$?

  3. Finally, in the derived result $\omega=\frac{2}{N}\sum_{i=1}^N Cov(\hat{y}_i,y_i)$, how does this covariance even make sense as I am assuming $y_i$ is the training label for data point $i$ in the train set $T$, so it should just be deterministic, no? Unless we are treating the train set as a random variable here?


1 Answer 1

  1. Yes, $\hat f$ is random because $\hat f$ is a function of the training set. The generalisation error is also a function of $T$: a 'good' training set might give you lower generalisation error

  2. The book defines $E_{\bf y}$ as the expectation over the outcome variable in the training set, fixing the predictors in the training set. If you're using version 2 of the book, it's after equation 7.20. If you're using version 1 you should look for version 2, which is free now. This section has a footnote Indeed, in the first edition of our book, this section wasn’t sufficiently clear [about what is fixed and what is random]

  3. Yes, the training set is treated as random here.

  • $\begingroup$ Thanks for the answer. I am still a little confused about point 2. Namely, $Err_{in}$ does not depend at all on the true outcome variable $y_i$'s, only the true (fixed) predictors $x_i$'s, and new variable $Y_i^0$'s, which are independent of all other randomness in the setting. So why is it not true that $\mathbb{E}_{\mathbf{y}}[Err_{in}]=Err_{in}$? Or am I misunderstanding something? $\endgroup$
    – Max
    Commented Dec 15, 2022 at 17:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.