5
$\begingroup$

Polynomial regression fits a non-linear model to the data. But as a statistical estimation problem it's still linear in the sense that the regression function $h\left(\Theta, X\right)$ is linear in the unknown parameters $\Theta$.

When we use polynomial regression we actually give our linear model additional features like $X^2$ or $XY$. But with the same success you can give your model features like $\log\left(X\right)$ or $\exp\left(X\right)$, and after that apply least squares. So you can fit any kind of curvature to your data.

My question is: Why does non-linear regression assume a more general hypothesis space of functions - one that encompasses the hypothesis space of functions that you can get with linear regression? I mean why do we think that non-linear regression can fit more types of curvatures to the data than linear regression if linear regression itself (e.g. with polynomial or logarithmic features) can fit any curvature to the data?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
7
$\begingroup$
  1. Model Parsimony

If you have a sine curve, you can approximate it to arbitrary accuracy with its series expansion.

I’d probably rather estimate the two parameters of $\mathbb E[y]= A\sin(Bx)$ than the many parameters in a long series expansion.

Note that, because the $B$ is inside the nonlinear sine function, you cannot create the estimated-frequency sine curve with a sine basis function; you would have to pick a $B$, rather than estimate it from the data.

  1. Interpretation

Parameters in the nonlinear equation can have interpretations of interest. In the above equation, $A$ is the amplitude and $B$ relates to the frequency. Perhaps you can wrestle with a long polynomial that approximates the sine curve in order to get at frequency and amplitude, but they are immediate from the nonlinear equation.

Improve this answer
$\endgroup$
12
  • $\begingroup$ @mathgeek: your question can be generalized even further: why use a non-linear function when any function can be approximated by a neural net ( proof is by Hornik et al in early 90's. I forget journal title ). So, why aren't neural nets used instead of non-linear functions ? Dave's answer still applies to this case, particularly the second part about interpretation. $\endgroup$
    – mlofton
    Oct 2, 2021 at 18:58
  • 3
    $\begingroup$ @mathgeek If you didn’t know the frequency $(B)$ of the sine curve, how would you fit to that with a linear equation? $\endgroup$
    – Dave
    Oct 2, 2021 at 19:04
  • 2
    $\begingroup$ @mathgeek How many polynomial terms do you need to get the Taylor series to equal the sine curve? Because of the potential to do series expansions, you are correct that linear regressions with nonlinear basis functions are going to be possible and will be able to approximate an awful lot of curves, but if you know (perhaps from your knowledge of the physics of how springs work) to expect $A\sin(Bx)$, then you might prefer to fit just those two parameters. $\endgroup$
    – Dave
    Oct 2, 2021 at 19:13
  • 2
    $\begingroup$ I want you to run a simulation in the software of your choosing where you simulate a relationship $y=\sin(2x)$ and then fit a linear regression with that nonlinear basis function $\sin(X)$. Plot the real scatter plot along with the predicted curve. Are you happy with the performance? $\endgroup$
    – Dave
    Oct 2, 2021 at 19:19
  • 2
    $\begingroup$ You might approximate it well if you take five or five-thousand terms of the Taylor expansion, but that puts you at risk of overfitting, far more than the two parameters of $A\sin(Bx)$ (if you know to expect a sine curve). Further, if you don’t restrict the Taylor expansion to be a polynomial regression on the appropriate monomial terms (sine and cosine take alternating terms of the exponential expansion, right?), then you would needlessly fit parameters that should be zero. $\endgroup$
    – Dave
    Oct 2, 2021 at 19:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.