I have a conceptual question about why (processing power/storage aside) would you ever just use a regular linear regression without adding polynomial features? It seems like adding polynomial features (without overfitting) would always produce better results? I know linear regression can fit more than just a line but that is only once you decide to add polynomial features correct? My experience with python using sklearn's libraries.
2
-
$\begingroup$ It depends on your model and data. Do you have a prior on your data? Do you know that it is linear or not? Do you want to extrapolate? $\endgroup$– DorianCommented Jun 24, 2020 at 16:59
-
2$\begingroup$ Polynomial regression is extremely dangerous for extrapolation. If you only want interpolation then other methods such as splines or generalized additive models can provide more flexibility than simple polynomials. $\endgroup$– HenryCommented Jun 30, 2020 at 14:15
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
At a minimum, you should consider cross-posting this to the Data Science stack exchange site (stats is more in tune with the statistical, ie math, underpinnings).
That said, there is a trade-off between variance and bias in pretty much every model of this sort. Essentially, there are a few factors here.
- Overfitting. You dismissed this with a handwave but it's one of the most critical aspects. A high bias model (fewer parameters in this case) helps guard against overfitting -- but at the cost of sensitivity.
- There is a sweet spot in increasing the complexity of the model. At some point, it might not improve the fit to the training data.
Additionally, that hand-wave of setting "processing power/storage aside" really is non-trivial as are the labor-hours involved.
I hope that helps but, again, this (and a number of questions like it) are better asked here.
$\begingroup$
$\endgroup$
There's a lot of nuance to this, as a general rule you chose the simpler model all things being equal. After building a polynomial model, you can F-test against a first-order model and go from there. Here is a link to more on this.
