7
$\begingroup$

James, Witten, Hastie, Tibshirani on page 35 of their book state with reference to the figure below:

In Figure 2.11, the true f [given by the black curve] is substantially non-linear, so no matter how many training observations we are given, it will not be possible to produce an accurate estimate using linear regression.

enter image description here

Is this statement correct? Keeping in mind the widespread misunderstanding that linear regression can only model linear relationships/missing the fact that linear regression refers to linearity in parameters and can accommodate a large collection of non-linearities, I wonder whether this statement is correct. And if it is, I wonder what it is about this particular shape of f that puts it out of reach for the non-linearities that can be accommodated by linear regression?

page 35:

"... linear regression assumes that there is a linear relationship between Y and $X_1$,$X_2$,...,$X_p$."

page 90:

"The linear regression model assumes that there is a straight-line relationship between the predictors and the response."

On page 91 they call the model in 3.36 a linear model, even with an exclamation mark.

page 91:

(3.36) $mpg=\beta_0+\beta_1*horsepower+\beta_2*horsepower^2+\epsilon$

Equation 3.36 involves predicting mpg using a non-linear function of horsepower. But it is still a linear model! That is, (3.36) is simply a multiple linear regression model with $X_1=horsepower$ and $X_2=horsepower^2$.

Improve this question
$\endgroup$
4
  • 5
    $\begingroup$ The full context answers your question: by "linear regression" in this context they mean linear in $X:$ "linear regression assumes that there is a linear relationship between Y and X1,X2, . . . ,Xp. It is unlikely that any real-life problem truly has such a simple linear relationship, and so performing linear regression will undoubtedly result in some bias in the estimate of f. In Figure 2.11, the true f is substantially non-linear,..." $\endgroup$
    – whuber ♦
    May 18 at 14:52
  • 2
    $\begingroup$ Thank you, whuber. I see. They changed the definition of linear regression here. I get it now but I would imagine this must have confused more people than just myself. $\endgroup$
    – ColorStatistics
    May 18 at 14:55
  • $\begingroup$ On page 63 the authors state in a footnote: "The assumption of linearity is often a useful working model. However, despite what many textbooks might tell us, we seldom believe that the true relationship is linear". Well..... first, they first reduced "linear regression" from a powerful tool that can accommodate non-linearities by assuming linearity with the variables rather than with the parameters... second, they then say that we should not believe the assumption of linearity, "contrary to what many books tell us"... the many books likely talk about linearity in parameters not in variables $\endgroup$
    – ColorStatistics
    May 18 at 17:02
  • 4
    $\begingroup$ Do you know the quote (I think attributed to Neumann): "With a degree 6 polynomial I can fit an elephant, with degree 7 I can make it wiggle its trunk." Polynomial regression is linear in its parameters but the statement that practically anything can be well fitted by a high degree polynomial is not useful in statistics (but it is true). $\endgroup$
    – quarague
    May 19 at 6:46

3 Answers 3

Reset to default
12
$\begingroup$

I wouldn't say the authors are wrong as such, but they weren't adequately careful with the wording. Often it's clear from context whether you mean simple or multiple linear regression, but here it wasn't.

The authors should have written

...it will not be possible to produce an accurate estimate using simple linear regression.

...where simple linear regression means using a single predictor: only $X_1$.

You are right that in this same paragraph, the authors had been discussing (multiple) linear regression in general:

For example, linear regression assumes that there is a linear relationship between $Y$ and $X_1$, $X_2$, ..., $X_p$.

There's no reason you couldn't define e.g. polynomials such as $X_2=X_1^2$ and $X_3=X_1^3$, in which case a linear model should indeed be able to fit that true $f$ reasonably well.

Edit: Alternately, to @whuber's point, they could have said

...it will not be possible to produce an accurate estimate using a regression that is linear in $X$.

Especially given that in Chapter 3 the authors note how a polynomial regression (nonlinear in $X$) is still a linear model (linear in the $\beta$s), it would have been helpful to be more explicit here. Their point here was not to say that the $f$ plotted in Figure 2.11 can't be well-approximated by a model linear in the $\beta$s---only that it can't be well-approximated by a model linear in $X$.

Improve this answer
$\endgroup$
7
  • 6
    $\begingroup$ I think this ignores, and is contrary to, the setting of the figure and the quotation. The paragraph begins "On the other hand, bias refers to the error that is introduced by approximating a real-life problem, which may be extremely complicated, by a much simpler model." Clearly, then, allowing for the manipulations you suggest is not to the point. The entire point is that trying to approximate a curve by a line doesn't work well, no matter how large a dataset you might have. $\endgroup$
    – whuber ♦
    May 18 at 15:13
  • 2
    $\begingroup$ @whuber I totally agree with you that the authors' point was "you can't approximate a curve well with a straight line, even with more data." But I agree with OP that their phrasing was legitimately confusing. I don't see them mentioning polynomials before this page 35, so this writeup is probably fine for introducing linear regr to a novice who's unlikely to think of "what about $X_2=X_1^2"? But it's unnecessarily ambiguous for someone who has seen polynomial models before -- & confusing for someone who reads Ch 3 (p.92 tells us a quadratic fit is a linear model!) and later reviews Ch 2. $\endgroup$
    – civilstat
    May 18 at 20:56
  • $\begingroup$ @whuber The OP asked if there's something special about this particular $f$ that means it can't be fit by multiple linear regression. There may be weird things like "$f(x)=1$ when $x$ is rational and $0$ otherwise" which inherently don't work well with MLR (though I guess even there in principle you could define an indicator "$x$ is rational"). But writing "simple lin reg" on p.35 would have made it unambiguously clear that this kind of weird $f$ wasn't the authors' point, especially in light of their own later emphasis on polynomial regression still being a linear model. $\endgroup$
    – civilstat
    May 18 at 21:00
  • $\begingroup$ @whuber ...that said, on re-reading your comment, just explicitly saying "linear in X" would have been another reasonable way to clarify the sentence that confused the OP. Thanks for pushing back on this; I've tried to incorporate it into the answer above. $\endgroup$
    – civilstat
    May 19 at 1:49
  • 1
    $\begingroup$ @ColorStatistics Alas, there's a lot of overloaded terms and notation in statistics! Among colleagues, a statistician might say "linear model" to mean linear in X in one place but linear in $\beta$ in another place, and neither is "wrong" if it's clear from context what they mean. I'd rather frame it as: it's pedagogically unsound to be ambiguous in an intro book. Either pick one definition and stick with it, OR (as this is an applied book, not a theory book) deliberately tell readers "Common usage might go either way, so let's help you learn how to tell from context..." $\endgroup$
    – civilstat
    May 19 at 13:06
7
$\begingroup$

A straight line is always going to leave something to be desired. In that sense, the claim is correct.

However, the true relationship looks cubic, which means that $\mathbb E[Y\vert X] = \beta_0 + \beta_1X + \beta_2X^2 + \beta_3X^3$ might be a reasonable model. Since this is linear in the parameters, the model is a linear regression, yet the fit should be reasonable. In that sense, the claim in incorrect.

Once you get into function convergence theorems in mathematical analysis, you will see that linear combinations of polynomial terms can get arbitrarily close to an awful lot of functions.

Improve this answer
$\endgroup$
0
7
$\begingroup$

Can (some) linear regression model this (population) function accurately?

Yes, it can with polynomial regression (see the plot below) which is a particular special case of linear regression. (Or from a particular viewpoint an extension of linear regression)

But, at that point in the book linear regression is to be interpreted as following. (Emphasis is mine in the following quote)

On the other hand, bias refers to the error that is introduced by approximating a real-life problem, which may be extremely complicated, by a much simpler model. For example, linear regression assumes that there is a linear relationship between Y and $X_1, X_2,\dots,X_p$. It is unlikely that any real-life problem truly has such a simple linear relationship, and so performing linear regression will undoubtedly result in some bias in the estimate of $f$.

Sure, you can have that the regression includes vectors in the regressor matrix that are a non-linear function of some variable, like fitting a polynomial or using log-transformed variables.

Later on in the book they are doing exactly this

Non-linear Relationships

As discussed previously, the linear regression model (3.19) assumes a linear relationship between the response and predictors. But in some cases, the true relationship between the response and the predictors may be non-linear. Here we present a very simple way to directly extend the linear model to accommodate non-linear relationships, using polynomial regression.

So linear regression can express relationships between variables that are not a straight line. Below is an example of fitting a cubic with a copy of the data from that figure.

But that is besides the point that is being made in the book, where they are explaining bias-variance trade off and the idea of using more flexible functions to estimate the distribution of some population.

example fit with cubic curve

data = matrix(c(0.3, 26.0, 2.9, 24.7, 4.4, 22.1, 7.6, 18.6, 14.0, 16.8, 14.9, 15.5, 15.2, 13.8, 17.8, 11.8, 19.0, 12.6, 21.9, 10.5, 22.4, 12.6, 25.9, 12.6, 26.8, 13.8, 30.6, 12.8, 33.5, 11.1, 34.4, 11.9, 35.6, 11.6, 37.0, 12.2, 37.9, 11.3, 39.9, 11.5, 40.2, 12.9, 42.9, 11.5, 44.0, 11.0, 45.2, 11.3, 46.9, 14.0, 47.5, 13.5, 48.4, 13.9, 49.3, 11.3, 49.6, 13.5, 51.9, 13.5, 53.1, 12.6, 57.7, 14.1, 59.5, 11.4, 64.4, 14.9, 67.1, 13.2, 71.1, 13.4, 71.4, 13.8, 71.7, 12.7, 72.3, 11.9, 76.1, 11.8, 76.7, 9.5, 78.7, 9.6, 82.8, 7.3, 83.4, 6.6, 87.2, 3.8, 90.7, 0.2, 92.1, 1.0, 92.7, -0.3, 94.2, -2.4, 99.1, -8.5), ncol = 2, byrow = TRUE)

colnames(data) = c("x","y")
data = as.data.frame(data)

plot(data$x,data$y)
mod = lm(y ~ x + I(x^2) + I(x^3), data = data)
lines(data$x, predict(mod))
summary(mod)$r.squared # returns 0.9715887

What is the definition of a linear model in James, Witten, Hastie, Tibshirani's book?

In the subscript of Figure 3.8 on page 91 they write "the linear regression fit is shown in orange. The linear regression fit for a model that includes horsepower² is shown as a blue curve." So for these authors the 'linear regression' is just simple linear regression unless you specify it differently. You may like it or not, but this is just subjective. In discussing the underlying topics you should get around this. The topics themselves don't care how they are being called.

Also, on page 289 the author's define polynomial regression explicitly as an extension of linear regression

Polynomial regression extends the linear model by adding extra predictors, obtained by raising each of the original predictors to a power. For example, a cubic regression uses three variables, $X, X^2 , and X^3$, as predictors. This approach provides a simple way to provide a non-linear fit to data.

In a certain sense polynomial regression is again a form of linear regression but with different/additional/more predictors. The original set of predictors has changed.

Where does the confusion stem from?

  • Strictly speaking linear regression is a linear model. It fits a linear function of the input variables to estimate the outcome variable. When you restrict linear regression to be defined as only a linear combination of the input variables, then linear regression can not model the cubic function accurately if $y$ is the output and $x$ is the input variable. Standard linear regression, which is uses a linear combination of the regressor variables, can not express non-linear relationships between the output variable and the regressor variables.
  • However, by adding non-linear transformations of the input/regressor variables as additional input/regressor variables we can use linear regression to model relationships between two variables that are non-linear. So we can have linear regression to express some non-linear relationships. But to do this we need additional input variables $x^2$ and $x^3$. This is different from plain vanilla linear regression and the authors consider this as an extension.

Linear regression can only model linear relationships between the response variable and the regressor variables. Polynomial regression is when we add additional regressors such that we are able to express a non-linear relationship between a response variable and an input variable. This polynomial regression is from a certain perspective a linear regression as well but only in terms of the new set of regressor variables which include the non-linear transformations of the original set of regressor variables.

Polynomial regression is expressing a non-linear relationship between the response $𝑦$ and regressor variable $𝑥$, but it is expressing a linear relationship between the response $𝑦$ and regressor variables $𝑥,𝑥^2,𝑥^3,\dots,𝑥^𝑝$. If your only regressor variable is $𝑥$ then you can not use linear regression to express the non-linear relationship.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ @SextusEmpiricus "it is a bit overly nitpicking to state that it is unclear what sort of wine is being referred to because it was not specified or that it is unclear if the term wine can mean different things in different places of a book" -- I'd agree in terms of a conversation among professionals! You know what they meant, because you're already an expert. But since this is marketed as an intro book, I'd rather put the burden on the book's authors to be unambiguous -- not on novice readers to understand the ambiguity right away. $\endgroup$
    – civilstat
    May 19 at 13:31
  • $\begingroup$ @civilstat they clearly state what they mean with a linear regression and the image is obvious that they go from a simple linear model to more complex spline. It is hairsplitting to focus on some definition of 'linear regression' as also being able to include polynomial models. If they would need to try to 'fix' this ambiguity in the definition then it would make the text more verbose and that introduces new problems. Then you get students that complain because the text is not intuitive and getting to the point but emphasizes the exact definition of the term 'linear regression'. $\endgroup$
    – Sextus Empiricus
    May 19 at 14:08
  • 1
    $\begingroup$ I re-read the text pages and your answer. Your answer helped me understand the authors' perspective, whether or not I agree with it. I will change the - to a + and will delete most of the comments above. Thank you. $\endgroup$
    – ColorStatistics
    May 19 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.