7
$\begingroup$

Page 24 of Introduction to Statistical Learning states:

...linear regression is a relatively inflexible approach, because it can only generate linear functions such as the lines shown in Figure 2.1 or the plane shown in Figure 2.3.

Later on, linear regression is defined more formally as assuming $f$ takes on form

$$ f(X) = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \ldots + \beta_p X_p $$

and "solving" for the $\beta_i$ using techniques like the "ordinary least squares" method (here I'm using the notation from the book).

Question: If this is really how linear regression is defined, why do we say that "linear regression [can]...only generate linear functions such as [lines and planes]"?

For example, if our model for $\hat{Y}$ is as follows:

$$ \hat{Y} = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \ldots + \beta_p X_p $$

then any (or all) of the $X_j$ could be themselves highly non-linear or otherwise pathological functions that take on many bizarre shapes that look nothing like lines or planes. That being the case, a linear combination of those pathological random variables could end up looking like something that fails to resemble a line or a plane. Doesn't this show the quote from above to be false?

Improve this question
$\endgroup$
5
  • 4
    $\begingroup$ Good question. A very closely related one at stats.stackexchange.com/questions/148638 concerns what "linear" in "linear model" might possibly mean. $\endgroup$
    – whuber ♦
    Apr 22, 2016 at 16:19
  • $\begingroup$ You may be also happy with this post, regarding the comment by @whuber on my initial (now erased) answer. $\endgroup$
    – Antoni Parellada
    Apr 22, 2016 at 17:16
  • $\begingroup$ @whuber: just to be clear, that post indicates that there are several notions of linearity, and that -- at the very least -- the quote above is wrong (in the sense that it is indeed possible for a model to be considered linear but in no way to resemble a line or a plane). Is my understanding correct here? $\endgroup$
    – George
    Apr 22, 2016 at 17:39
  • 1
    $\begingroup$ I believe your understanding is correct, but I wouldn't go so far as to characterize the quote as "wrong": if you look at it the right way, such a linear regression indeed only fits lines, planes, and other linear subspaces. I think the fundamental point is that you--the modeler--begin by establishing a mapping from a space of possible regressor values into a space of regressors (such as mapping numbers $x$ to vectors $(1,x,x^2)$). Although that mapping may be nonlinear, the model in terms of the image vectors is perfectly linear. $\endgroup$
    – whuber ♦
    Apr 22, 2016 at 18:14
  • $\begingroup$ The data can all be non-linear but when you run linear regression or perform least squares you can be sure that you'll get a line. Your model for Y hat leaves off the residuals. All of that non linearity in the predictors you discussed will just be part of the residuals around the fitted line after OLS. The line may not describe much of the variance, but it's still a line. $\endgroup$
    – conv3d
    Apr 23, 2016 at 9:10

1 Answer 1

Reset to default
1
$\begingroup$

Unfortunatelly my rating is too low to add a comment, that's why I have to answer.

Just imagine that there is some feature $Z$. $X_1 = \sin Z, \ X_2 = \cos Z$, and $f(X) = \beta_0 + \beta_1 X_1 + \beta_2 X_2 = \beta_0 + \beta_1 \sin Z + \beta_2 \cos Z$.

Will $f(X)$ be linear if the coordinate system is $(Y, Z)$? And what about $(Y, X_1, X_2)$?

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.