1
$\begingroup$

So the definition of linear regression is that the response variable is a linear function of the estimators. If we consider univariate regression (for ease of visualization), we have $$ y = \beta_1x + \beta_0 $$ But we could also have $$ y = \beta_1x^2 + \beta_0 \\ y = \beta_1 \exp(\log(x^3)) $$ which also satisfies the condition that the response variable is a linear function of the ESTIMATORs.

I find this terminology to be a bit confusing as I would expect "linear" regression to be restricted to strict lines in univariate regression.

When linear regression is introduced in courses, the examples are always straight lines, and I think some instructors even introduce linear regression as fitting a LINEAR line to a set of data, but that's not true.

So isn't it rather confusing that it's called "linear" regression? I feel like "linear" regression connotes that the fit will be a straight line (in the univariate case).

$\endgroup$
6
  • $\begingroup$ I've personally thought (I'm not a statistician), that if we don't change the parameter, and by using a substitution of variables can make it linear and solve it with least squares, then it is linear. A rational would be a problem but in this if $z = x^2$ then $y\left(z\right)$ is linear in z. $\endgroup$ Jun 25 '20 at 18:47
  • 2
    $\begingroup$ Contrast $y = \beta_0 + \beta_1 x$ (linear in parameters) with $y = b_0 + x^{\beta_1}$ or $y = \beta_0 + \beta_{1}^{x}$ (nonlinear in parameters). $\endgroup$
    – Alexis
    Jun 25 '20 at 20:23
  • $\begingroup$ @EngrStudent Right, that's how I was thinking about it just now. If we consider something like $y = sin(x)$. We could define the transformation $z = sin(x)$. Then $y=z$ is linear in the connotated sense of the word. $\endgroup$
    – 24n8
    Jun 25 '20 at 20:27
  • $\begingroup$ I like how MathematicalMonk addresses this: youtube.com/watch?v=rVviNyIR-fI. “It’s not just lines & planes.” $\endgroup$
    – Dave
    Jun 26 '20 at 13:50
  • 2
    $\begingroup$ The sense in which regressing $y$ on nonlinear transformations of $x$ still is just a matter of lines, planes, and linear subspaces generally is explained and illustrated at stats.stackexchange.com/a/354256/919. $\endgroup$
    – whuber
    Jun 26 '20 at 15:38
7
$\begingroup$

Check out these threads:

How to tell the difference between linear and non-linear regression models?

Why must linear regressions only generate linear functions that resemble "lines or planes" (*Introduction to Statistical Learning* question)?

"Linear" is talking about the calculation that happens to the unknown coefficients $\beta$. Any regression of the form

$$y \approx \beta^Tf(x)$$

is reasonably called linear as long as $f$ and $x$ are known. This is because a lot of the same theory and computation applies when you go to estimate $\beta$, regardless of what values $f$ and $x$ take.

EDIT: When might you know $f$?

  • Spline regression
  • RBF's or Nardaraya-Watson as in PRML Section 6.3 (This is like an interaction term on steroids; it's completely nonparametric. It's still linear in $\beta$.)
  • When using a wavelet basis

How do you know if you should be using a wavelet basis or a spline? Sorry, that's a whole different question.

$\endgroup$
5
  • $\begingroup$ "linear" means no interaction terms. $\endgroup$
    – LBogaardt
    Jun 25 '20 at 18:52
  • $\begingroup$ @LBogaardt When I think of interaction, I think of interaction between $\beta_i$ and $\beta_j$ for $i \neq j$. But $\beta_i^2$ would no longer be linear. $\endgroup$
    – 24n8
    Jun 25 '20 at 19:00
  • $\begingroup$ As far as I know, $\beta_i x_i + \beta_j x_j$ is called 'linear', so is $\beta_i log(x_i) + \beta_j x_j^5$, but $\beta_{ij} x_i x_j$ is not. $\endgroup$
    – LBogaardt
    Jun 25 '20 at 19:35
  • 6
    $\begingroup$ @LBogaardt The latter should still be within the definition of linear regression because $y$ is still a linear function of $\beta$. You can introduce a new variable $z = x_ix_j$. It's just $z$ will have a strong degree of multicollinearity with $x_i$ and $x_j$. $\endgroup$
    – 24n8
    Jun 25 '20 at 19:51
  • 1
    $\begingroup$ When you're given an input variable, in this case, $f(x)$, which situations would you ever know what $f(x)$ is? $\endgroup$
    – 24n8
    Jun 25 '20 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.