2
$\begingroup$

So the definition of linear regression is that the response variable is a linear function of the estimators. If we consider univariate regression (for ease of visualization), we have $$ y = \beta_1x + \beta_0 $$ But we could also have $$ y = \beta_1x^2 + \beta_0 \\ y = \beta_1 \exp(\log(x^3)) $$ which also satisfies the condition that the response variable is a linear function of the ESTIMATORs.

I find this terminology to be a bit confusing as I would expect "linear" regression to be restricted to strict lines in univariate regression.

When linear regression is introduced in courses, the examples are always straight lines, and I think some instructors even introduce linear regression as fitting a LINEAR line to a set of data, but that's not true.

So isn't it rather confusing that it's called "linear" regression? I feel like "linear" regression connotes that the fit will be a straight line (in the univariate case).

Improve this question
$\endgroup$
6
  • $\begingroup$ I've personally thought (I'm not a statistician), that if we don't change the parameter, and by using a substitution of variables can make it linear and solve it with least squares, then it is linear. A rational would be a problem but in this if $z = x^2$ then $y\left(z\right)$ is linear in z. $\endgroup$
    – EngrStudent
    Jun 25, 2020 at 18:47
  • 2
    $\begingroup$ Contrast $y = \beta_0 + \beta_1 x$ (linear in parameters) with $y = b_0 + x^{\beta_1}$ or $y = \beta_0 + \beta_{1}^{x}$ (nonlinear in parameters). $\endgroup$
    – Alexis
    Jun 25, 2020 at 20:23
  • $\begingroup$ @EngrStudent Right, that's how I was thinking about it just now. If we consider something like $y = sin(x)$. We could define the transformation $z = sin(x)$. Then $y=z$ is linear in the connotated sense of the word. $\endgroup$
    – 24n8
    Jun 25, 2020 at 20:27
  • $\begingroup$ I like how MathematicalMonk addresses this: youtube.com/watch?v=rVviNyIR-fI. “It’s not just lines & planes.” $\endgroup$
    – Dave
    Jun 26, 2020 at 13:50
  • 2
    $\begingroup$ The sense in which regressing $y$ on nonlinear transformations of $x$ still is just a matter of lines, planes, and linear subspaces generally is explained and illustrated at stats.stackexchange.com/a/354256/919. $\endgroup$
    – whuber ♦
    Jun 26, 2020 at 15:38

1 Answer 1

Reset to default
8
$\begingroup$

Check out these threads:

How to tell the difference between linear and non-linear regression models?

Why must linear regressions only generate linear functions that resemble "lines or planes" (*Introduction to Statistical Learning* question)?

"Linear" is talking about the calculation that happens to the unknown coefficients $\beta$. Any regression of the form

$$y \approx \beta^Tf(x)$$

is reasonably called linear as long as $f$ and $x$ are known. This is because a lot of the same theory and computation applies when you go to estimate $\beta$, regardless of what values $f$ and $x$ take.

EDIT: When might you know $f$?

  • Spline regression
  • RBF's or Nardaraya-Watson as in PRML Section 6.3 (This is like an interaction term on steroids; it's completely nonparametric. It's still linear in $\beta$.)
  • When using a wavelet basis

How do you know if you should be using a wavelet basis or a spline? Sorry, that's a whole different question.

Improve this answer
$\endgroup$
5
  • $\begingroup$ "linear" means no interaction terms. $\endgroup$
    – LBogaardt
    Jun 25, 2020 at 18:52
  • $\begingroup$ @LBogaardt When I think of interaction, I think of interaction between $\beta_i$ and $\beta_j$ for $i \neq j$. But $\beta_i^2$ would no longer be linear. $\endgroup$
    – 24n8
    Jun 25, 2020 at 19:00
  • $\begingroup$ As far as I know, $\beta_i x_i + \beta_j x_j$ is called 'linear', so is $\beta_i log(x_i) + \beta_j x_j^5$, but $\beta_{ij} x_i x_j$ is not. $\endgroup$
    – LBogaardt
    Jun 25, 2020 at 19:35
  • 6
    $\begingroup$ @LBogaardt The latter should still be within the definition of linear regression because $y$ is still a linear function of $\beta$. You can introduce a new variable $z = x_ix_j$. It's just $z$ will have a strong degree of multicollinearity with $x_i$ and $x_j$. $\endgroup$
    – 24n8
    Jun 25, 2020 at 19:51
  • 1
    $\begingroup$ When you're given an input variable, in this case, $f(x)$, which situations would you ever know what $f(x)$ is? $\endgroup$
    – 24n8
    Jun 25, 2020 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.