Why can 'linear' regressions include non-linear transformations of the independent variables?

So the definition of linear regression is that the response variable is a linear function of the estimators. If we consider univariate regression (for ease of visualization), we have $$y = \beta_1x + \beta_0$$ But we could also have $$y = \beta_1x^2 + \beta_0 \\ y = \beta_1 \exp(\log(x^3))$$ which also satisfies the condition that the response variable is a linear function of the ESTIMATORs.

I find this terminology to be a bit confusing as I would expect "linear" regression to be restricted to strict lines in univariate regression.

When linear regression is introduced in courses, the examples are always straight lines, and I think some instructors even introduce linear regression as fitting a LINEAR line to a set of data, but that's not true.

So isn't it rather confusing that it's called "linear" regression? I feel like "linear" regression connotes that the fit will be a straight line (in the univariate case).

• I've personally thought (I'm not a statistician), that if we don't change the parameter, and by using a substitution of variables can make it linear and solve it with least squares, then it is linear. A rational would be a problem but in this if $z = x^2$ then $y\left(z\right)$ is linear in z. Jun 25 '20 at 18:47
• Contrast $y = \beta_0 + \beta_1 x$ (linear in parameters) with $y = b_0 + x^{\beta_1}$ or $y = \beta_0 + \beta_{1}^{x}$ (nonlinear in parameters). Jun 25 '20 at 20:23
• @EngrStudent Right, that's how I was thinking about it just now. If we consider something like $y = sin(x)$. We could define the transformation $z = sin(x)$. Then $y=z$ is linear in the connotated sense of the word.
– 24n8
Jun 25 '20 at 20:27
• I like how MathematicalMonk addresses this: youtube.com/watch?v=rVviNyIR-fI. “It’s not just lines & planes.”
– Dave
Jun 26 '20 at 13:50
• The sense in which regressing $y$ on nonlinear transformations of $x$ still is just a matter of lines, planes, and linear subspaces generally is explained and illustrated at stats.stackexchange.com/a/354256/919.
– whuber
Jun 26 '20 at 15:38

How to tell the difference between linear and non-linear regression models?

Why must linear regressions only generate linear functions that resemble "lines or planes" (*Introduction to Statistical Learning* question)?

"Linear" is talking about the calculation that happens to the unknown coefficients $$\beta$$. Any regression of the form

$$y \approx \beta^Tf(x)$$

is reasonably called linear as long as $$f$$ and $$x$$ are known. This is because a lot of the same theory and computation applies when you go to estimate $$\beta$$, regardless of what values $$f$$ and $$x$$ take.

EDIT: When might you know $$f$$?

• Spline regression
• RBF's or Nardaraya-Watson as in PRML Section 6.3 (This is like an interaction term on steroids; it's completely nonparametric. It's still linear in $$\beta$$.)
• When using a wavelet basis

How do you know if you should be using a wavelet basis or a spline? Sorry, that's a whole different question.

• "linear" means no interaction terms. Jun 25 '20 at 18:52
• @LBogaardt When I think of interaction, I think of interaction between $\beta_i$ and $\beta_j$ for $i \neq j$. But $\beta_i^2$ would no longer be linear.
– 24n8
Jun 25 '20 at 19:00
• As far as I know, $\beta_i x_i + \beta_j x_j$ is called 'linear', so is $\beta_i log(x_i) + \beta_j x_j^5$, but $\beta_{ij} x_i x_j$ is not. Jun 25 '20 at 19:35
• @LBogaardt The latter should still be within the definition of linear regression because $y$ is still a linear function of $\beta$. You can introduce a new variable $z = x_ix_j$. It's just $z$ will have a strong degree of multicollinearity with $x_i$ and $x_j$.
– 24n8
Jun 25 '20 at 19:51
• When you're given an input variable, in this case, $f(x)$, which situations would you ever know what $f(x)$ is?
– 24n8
Jun 25 '20 at 21:09