I have a conceptual question about why (processing power/storage aside) would you ever just use a regular linear regression without adding polynomial features? It seems like adding polynomial features (without overfitting) would always produce better results? I know linear regression can fit more than just a line but that is only once you decide to add polynomial features correct? My experience with python using sklearn's libraries.
At a minimum, you should consider cross-posting this to the Data Science stack exchange site (stats is more in tune with the statistical, ie math, underpinnings).
That said, there is a trade-off between variance and bias in pretty much every model of this sort. Essentially, there are a few factors here.
- Overfitting. You dismissed this with a handwave but it's one of the most critical aspects. A high bias model (fewer parameters in this case) helps guard against overfitting -- but at the cost of sensitivity.
- There is a sweet spot in increasing the complexity of the model. At some point, it might not improve the fit to the training data.
Additionally, that hand-wave of setting "processing power/storage aside" really is non-trivial as are the labor-hours involved.
I hope that helps but, again, this (and a number of questions like it) are better asked here.
There's a lot of nuance to this, as a general rule you chose the simpler model all things being equal. After building a polynomial model, you can F-test against a first-order model and go from there. Here is a link to more on this.