Rigorous statement of expectations for the bias-variance trade-off

Question

Consider a data generating process $$Y=f(X)+\varepsilon$$ where $\varepsilon$ is independent of $x$ with $\mathbb E(\varepsilon)=0$ and $\text{Var}(\varepsilon)=\sigma^2_\varepsilon$. According to Hastie et al. "The Elements of Statistical Learning" (2nd edition, 2009) Section 7.3 p. 223, we can derive an expression for the expected prediction error of a regression fit $\hat f(X)$ at an input point $X=x_0$, using squared-error loss:

\begin{align} \text{Err}(x_0) &=\mathbb E[(Y-\hat f(x_0))^2|X=x_0]\\ &=(\mathbb E[\hat f(x_0)−f(x_0)])^2+\mathbb E[(\hat f(x_0)−\mathbb E[\hat f(x_0)])^2]+\sigma^2_\varepsilon\\ &=\text{Bias}^2\ \ \ \quad\quad\quad\quad\quad\;\;+\text{Variance } \quad\quad\quad\quad\quad\quad+ \text{ Irreducible Error} \end{align}

(where I use the notation $\text{Bias}^2$ instead of $\text{Bias}$).

Question: What are the expectations taken over? What is held fixed and what is random?

_{The question arose in the comments of the thread "Why is there a bias variance tradeoff? A counterexample".}

Answer 1

$X$ is assumed fixed; see Section 2.9, p. 37:

For simplicity here we assume that the values of $x_i$ in the sample are fixed in advance (nonrandom).

Then the only source of random variation here is $\varepsilon$. Hence, the expectations are taken w.r.t. to the distribution of $\varepsilon$.