Can (some) linear regression model this (population) function accurately?
Yes, it can with polynomial regression (see the plot below) which is a particular special case of linear regression. (Or from a particular viewpoint an extension of linear regression)
But, at that point in the book linear regression is to be interpreted as following. (Emphasis is mine in the following quote)
On the other hand, bias refers to the error that is introduced by approximating a real-life problem, which may be extremely complicated, by a much simpler model. For example, linear regression assumes that there is a linear relationship between Y and $X_1, X_2,\dots,X_p$. It is unlikely that any real-life problem truly has such a simple linear relationship, and so performing linear regression will undoubtedly result in some bias in the estimate of $f$.
Sure, you can have that the regression includes vectors in the regressor matrix that are a non-linear function of some variable, like fitting a polynomial or using log-transformed variables.
Later on in the book they are doing exactly this
As discussed previously, the linear regression model (3.19) assumes a linear relationship between the response and predictors. But in some cases, the true relationship between the response and the predictors may be non-linear. Here we present a very simple way to directly extend the linear model to accommodate non-linear relationships, using polynomial regression.
So linear regression can express relationships between variables that are not a straight line. Below is an example of fitting a cubic with a copy of the data from that figure.
But that is besides the point that is being made in the book, where they are explaining bias-variance trade off and the idea of using more flexible functions to estimate the distribution of some population.
data = matrix(c(0.3, 26.0, 2.9, 24.7, 4.4, 22.1, 7.6, 18.6, 14.0, 16.8, 14.9, 15.5, 15.2, 13.8, 17.8, 11.8, 19.0, 12.6, 21.9, 10.5, 22.4, 12.6, 25.9, 12.6, 26.8, 13.8, 30.6, 12.8, 33.5, 11.1, 34.4, 11.9, 35.6, 11.6, 37.0, 12.2, 37.9, 11.3, 39.9, 11.5, 40.2, 12.9, 42.9, 11.5, 44.0, 11.0, 45.2, 11.3, 46.9, 14.0, 47.5, 13.5, 48.4, 13.9, 49.3, 11.3, 49.6, 13.5, 51.9, 13.5, 53.1, 12.6, 57.7, 14.1, 59.5, 11.4, 64.4, 14.9, 67.1, 13.2, 71.1, 13.4, 71.4, 13.8, 71.7, 12.7, 72.3, 11.9, 76.1, 11.8, 76.7, 9.5, 78.7, 9.6, 82.8, 7.3, 83.4, 6.6, 87.2, 3.8, 90.7, 0.2, 92.1, 1.0, 92.7, -0.3, 94.2, -2.4, 99.1, -8.5), ncol = 2, byrow = TRUE)
colnames(data) = c("x","y")
data = as.data.frame(data)
mod = lm(y ~ x + I(x^2) + I(x^3), data = data)
summary(mod)$r.squared # returns 0.9715887
What is the definition of a linear model in James, Witten, Hastie, Tibshirani's book?
In the subscript of Figure 3.8 on page 91 they write "the linear regression fit is shown in orange. The linear regression fit for a model that includes horsepower² is shown as a blue curve." So for these authors the 'linear regression' is just simple linear regression unless you specify it differently. You may like it or not, but this is just subjective. In discussing the underlying topics you should get around this. The topics themselves don't care how they are being called.
Also, on page 289 the author's define polynomial regression explicitly as an extension of linear regression
Polynomial regression extends the linear model by adding extra predictors, obtained by raising each of the original predictors to a power. For example, a cubic regression uses three variables, $X, X^2 , and X^3$, as predictors. This approach provides a simple way to provide a non-linear fit to data.
In a certain sense polynomial regression is again a form of linear regression but with different/additional/more predictors. The original set of predictors has changed.
Where does the confusion stem from?
- Strictly speaking linear regression is a linear model. It fits a linear function of the input variables to estimate the outcome variable. When you restrict linear regression to be defined as only a linear combination of the input variables, then linear regression can not model the cubic function accurately if $y$ is the output and $x$ is the input variable. Standard linear regression, which is uses a linear combination of the regressor variables, can not express non-linear relationships between the output variable and the regressor variables.
- However, by adding non-linear transformations of the input/regressor variables as additional input/regressor variables we can use linear regression to model relationships between two variables that are non-linear. So we can have linear regression to express some non-linear relationships. But to do this we need additional input variables $x^2$ and $x^3$. This is different from plain vanilla linear regression and the authors consider this as an extension.
Linear regression can only model linear relationships between the response variable and the regressor variables. Polynomial regression is when we add additional regressors such that we are able to express a non-linear relationship between a response variable and an input variable. This polynomial regression is from a certain perspective a linear regression as well but only in terms of the new set of regressor variables which include the non-linear transformations of the original set of regressor variables.
Polynomial regression is expressing a non-linear relationship between the response $𝑦$ and regressor variable $𝑥$, but it is expressing a linear relationship between the response $𝑦$ and regressor variables $𝑥,𝑥^2,𝑥^3,\dots,𝑥^𝑝$. If your only regressor variable is $𝑥$ then you can not use linear regression to express the non-linear relationship.