Consider a data generating process $$Y=f(X)+\varepsilon$$ where $\varepsilon$ is independent of $x$ with $\mathbb E(\varepsilon)=0$ and $\text{Var}(\varepsilon)=\sigma^2_\varepsilon$. According to Hastie et al. "The Elements of Statistical Learning" (2nd edition, 2009) Section 7.3 p. 223, we can derive an expression for the expected prediction error of a regression fit $\hat f(X)$ at an input point $X=x_0$, using squared-error loss:

\begin{align} \text{Err}(x_0) &=\mathbb E[(Y-\hat f(x_0))^2|X=x_0]\\ &=(\mathbb E[\hat f(x_0)−f(x_0)])^2+\mathbb E[(\hat f(x_0)−\mathbb E[\hat f(x_0)])^2]+\sigma^2_\varepsilon\\ &=\text{Bias}^2\ \ \ \quad\quad\quad\quad\quad\;\;+\text{Variance } \quad\quad\quad\quad\quad\quad+ \text{ Irreducible Error} \end{align}

(where I use the notation $\text{Bias}^2$ instead of $\text{Bias}$).

Question: What are the expectations taken over? What is held fixed and what is random?

The question arose in the comments of the thread "Why is there a bias variance tradeoff? A counterexample".

  • $\begingroup$ $x_0$ and $f(x_0)$ are assumed fixed. $\hat{f}(.)$, however, depends on the training data, which is taken as random. Finally, there is randomness in $Y|X=x_0$, which is assumed independent of $\hat{f}(.)$ (conditional on $X=x_0$). $\endgroup$ – Tim Mak Jun 9 '20 at 10:01
  • $\begingroup$ @TimMak, sounds reasonable. So what are the expectations taken over? The random sampling of the training set jointly with the randomness in $\varepsilon$, i.e. a double integral? I have added the qualification that $\varepsilon$ is independent of $x$, but should that be the case? (I guess it should.) $\endgroup$ – Richard Hardy Jun 9 '20 at 10:16
  • $\begingroup$ Actually, I read the book in more detail and found that in their presentation $X$ is actually assumed fixed. Section 7.3 refers back I think, to Section 2.9, where they said "For simplicity here we assume that the values of $x_i$ in the sample are fixed in advance (nonrandom)". However, in other presentations, it is common to have $X$ assumed random also. Either way, $\hat{f}(.)$ depends on both $X$ and $Y$, and hence is random even when $X$ is held fixed. $\endgroup$ – Tim Mak Jun 10 '20 at 2:04
  • $\begingroup$ @TimMak, since you are knowledgeable about the matter, consider writing up an answer. $\endgroup$ – Richard Hardy Jun 10 '20 at 5:52

$X$ is assumed fixed; see Section 2.9, p. 37:

For simplicity here we assume that the values of $x_i$ in the sample are fixed in advance (nonrandom).

Then the only source of random variation here is $\varepsilon$. Hence, the expectations are taken w.r.t. to the distribution of $\varepsilon$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.