2
$\begingroup$

I'm well aware that when we use the expression "linear model" we are actually making reference to models that are linear on the parameters $\beta$. And because of that any polynomial regression will be consider as linear.

Anyway I've been reading "Elements of Statistical Learning" (Hastie, et.al.) and I found a phrase that confused me a little.

Linear regression, linear discriminant analysis, logistic regression and separating hyperplanes all rely on a linear model. It is extremely unlikely that the true function f(X) is actually linear in X. In regression problems, f(X) = E(Y |X) will typically be nonlinear and nonadditive in X, and representing f(X) by a linear model is usually a convenient, and sometimes a necessary, approximation. (p.139)

He states that generally we use, for the sake of simplicity, linear models, but that it's extremely unlikely to have a true function linear in X. And because of that we could use "non linear models" (but non linear in X, like for example polynomial regression).

In my opinion it's a little bit confusing, given that generally we use the terminology "non linear models" to talk about "non linear in parameters".

Am I missing something? Is it usual to use such terminology when referring to polynomial regression?

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
4
$\begingroup$

TL;DR The wording may be confusing, but he means that $f(X)$ is a linear function of $X$.

By linear model we mean approximating the relation between $Y$ and $X$ using a function of $X$, i.e. $Y \approx f(X)$, where the function $f$ is a linear function. Linear function is defined as $f(x) = a + bx$, where $a,b$ are parameters. If the model takes form of a non-linear function, then we call it a non-linear model. Notice that he discusses any machine learning model, not only regression models, and such models can be non-parametric, so "linearity in parameters" does not apply to them at all.

See also the GLMs must be 'linear in the parameters' and How to tell the difference between linear and non-linear regression models? threads.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I wasn't aware of that use of terminology, I've always believed that "non-linear" regression was referred to "non-linear in parameters" only. Thanks for the extra links. $\endgroup$
    – Chaos
    Jan 11, 2019 at 8:48
  • 1
    $\begingroup$ @RamiroScorolli statistical literature in many cases by "models" means regression models, while in machine learning literature this is much broader. For regression we usually talk about linearity in parameters. (Also see my small edit.) $\endgroup$
    – Tim ♦
    Jan 11, 2019 at 9:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.