Bias in bias-variance trade-off: how well the model can possibly approximate the DGP?

Question

Consider a data generating process $$Y=f(X)+\varepsilon$$ where $\varepsilon$ is independent of $x$ with $\mathbb E(\varepsilon)=0$ and $\text{Var}(\varepsilon)=\sigma^2_\varepsilon$. According to Hastie et al. "The Elements of Statistical Learning" (2nd edition, 2009) Section 7.3 p. 223, we can derive an expression for the expected prediction error of a regression fit $\hat f(X)$ at an input point $X=x_0$, using squared-error loss:

\begin{align} \text{Err}(x_0) &=\mathbb E[(Y-\hat f(x_0))^2|X=x_0]\\ &=(\mathbb E[\hat f(x_0)−f(x_0)])^2+\mathbb E[(\hat f(x_0)−\mathbb E[\hat f(x_0)])^2]+\sigma^2_\varepsilon\\ &=\text{Bias}^2\ \ \ \quad\quad\quad\quad\quad\;\;+\text{Variance } \quad\quad\quad\quad\quad\quad+ \text{ Irreducible Error} \end{align}

I am trying to understand bias. James et al. "Introduction to of Statistical Learning" (2nd edition, 2021) Section 2.2.2, concretely 2nd paragraph of p. 35 suggests that bias concerns how well the model could approximate the data generating process given infinite training data:

[B]ias refers to the error that is introduced by approximating a real-life problem, which may be extremely complicated, by a much simpler model. For example, linear regression assumes that there is a linear relationship between $Y$ and $X_1, X_2, \dots, X_p$. It is unlikely that any real-life problem truly has such a simple linear relationship, and so performing linear regression will undoubtedly result in some bias in the estimate of $f$. In Figure 2.11, the true $f$ is substantially non-linear, soo matter how many training observations we are given, it will not be possible to produce an accurate estimate using linear regression. In other words, linear regression results in high bias in this example.

Is such interpretation of bias correct?

Answer 1

This is not entirely precise. Consider the zero-mean AR(1) example in the thread "Bias-variance trade-off in case of biased estimators: is the bias zero?". When the model is estimated by conditional least squares (CLS), the estimated slope coefficient $\hat\varphi^{CLS}$ is biased but consistent. (For the record, the same could also be said about conditional and full maximum likelihood estimators; see links in the linked thread.) That means, there is a nonzero finite-sample bias that will be shrinking with the sample size until it becomes zero asymptotically. Therefore, the bias of a prediction from the model estimated by CLS will be specific to the sample size. (More generally, bias depends on the estimation method in addition to the model, something you do not immediately see from the quote.) In this sense, one cannot interpret bias as how well the model could approximate the data generating process given infinite training data.