4
$\begingroup$

Consider a data generating process $$Y=f(X)+\varepsilon$$ where $\varepsilon$ is independent of $x$ with $\mathbb E(\varepsilon)=0$ and $\text{Var}(\varepsilon)=\sigma^2_\varepsilon$. According to Hastie et al. "The Elements of Statistical Learning" (2nd edition, 2009) Section 7.3 p. 223, we can derive an expression for the expected prediction error of a regression fit $\hat f(X)$ at an input point $X=x_0$, using squared-error loss:

\begin{align} \text{Err}(x_0) &=\mathbb E[(Y-\hat f(x_0))^2|X=x_0]\\ &=(\mathbb E[\hat f(x_0)−f(x_0)])^2+\mathbb E[(\hat f(x_0)−\mathbb E[\hat f(x_0)])^2]+\sigma^2_\varepsilon\\ &=\text{Bias}^2\ \ \ \quad\quad\quad\quad\quad\;\;+\text{Variance } \quad\quad\quad\quad\quad\quad+ \text{ Irreducible Error} \end{align}

(where I use the notation $\text{Bias}^2$ instead of $\text{Bias}$).

Question: What are the expectations taken over? What is held fixed and what is random?

The question arose in the comments of the thread "Why is there a bias variance tradeoff? A counterexample".

$\endgroup$
5
  • $\begingroup$ $x_0$ and $f(x_0)$ are assumed fixed. $\hat{f}(.)$, however, depends on the training data, which is taken as random. Finally, there is randomness in $Y|X=x_0$, which is assumed independent of $\hat{f}(.)$ (conditional on $X=x_0$). $\endgroup$
    – Tim Mak
    Jun 9, 2020 at 10:01
  • $\begingroup$ @TimMak, sounds reasonable. So what are the expectations taken over? The random sampling of the training set jointly with the randomness in $\varepsilon$, i.e. a double integral? I have added the qualification that $\varepsilon$ is independent of $x$, but should that be the case? (I guess it should.) $\endgroup$ Jun 9, 2020 at 10:16
  • $\begingroup$ Actually, I read the book in more detail and found that in their presentation $X$ is actually assumed fixed. Section 7.3 refers back I think, to Section 2.9, where they said "For simplicity here we assume that the values of $x_i$ in the sample are fixed in advance (nonrandom)". However, in other presentations, it is common to have $X$ assumed random also. Either way, $\hat{f}(.)$ depends on both $X$ and $Y$, and hence is random even when $X$ is held fixed. $\endgroup$
    – Tim Mak
    Jun 10, 2020 at 2:04
  • $\begingroup$ @TimMak, since you are knowledgeable about the matter, consider writing up an answer. $\endgroup$ Jun 10, 2020 at 5:52
  • $\begingroup$ cs.cornell.edu/courses/cs4780/2018fa/lectures/… - best explanation I found $\endgroup$ Aug 12 at 5:55

1 Answer 1

1
$\begingroup$

$X$ is assumed fixed; see Section 2.9, p. 37:

For simplicity here we assume that the values of $x_i$ in the sample are fixed in advance (nonrandom).

Then the only source of random variation here is $\varepsilon$. Hence, the expectations are taken w.r.t. to the distribution of $\varepsilon$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.