So we use polynomial regression for estimating conditional mean functions of unknown form, and especially higher order polynomials are flexible. I've learned in the past that especially higher-order polynomials have undesirable effects when being fitted: values of the response variable for some special segments of the predictor will affect badly estimated values for other segments of the predictor. I would be grateful for any further explanation on this matter!

  • $\begingroup$ "I would be grateful for any further explanation on this matter!" is too vague for a really good answer. Please try to ask specific questions. $\endgroup$ – Glen_b -Reinstate Monica Jun 24 '17 at 4:18
  • $\begingroup$ Yes, sorry! Old professor told me this once, and I could not remember exactly what he said. $\endgroup$ – Erosennin Jun 24 '17 at 8:31

Think about how polynomials behave. Graph any polynomial. Look at what happens when you extrapolate beyond the data. The function will go to negative or positive infinity. That is almost always undesirable. But if you only need to estimate in the range where you already have data, then it should be OK, usually.

  • $\begingroup$ Aha, I get it. Of course! $\endgroup$ – Erosennin Jun 23 '17 at 11:57
  • 2
    $\begingroup$ How about spurious oscillations ... even for interpolation? If you answer were true, regression splines and smoothing splines would be pretty much out of business, or never gotten into business. $\endgroup$ – Mark L. Stone Jun 23 '17 at 12:19

Polynomials are notorious for undesirable non-local effects and aside from very low degree polynomials (2 or 3 in particular situations) are usually avoided for that reason. Peter's discussion gives some sense of why it happens, but in any case it's easy to observe in practice -- a simple case is to fit a high order polynomial to a few data points (e.g. a degree 6 polynomial to 9 data points say) and then notice the way the behavior of the whole fit alters as you change one of the points.

Splines are often used when there's no specific functional relationship in mind. For example, a similar situation when fitting natural cubic regression splines will have local effects - the fit for nearby points will change - but as you move more than a few knots away from the changing value, the fit doesn't alter. You can also parameterize your spline fit (via B-splines say) so the coefficients of the components are only affected locally.

Similarly, local regression methods (such as local linear regression via a bounded kernel supplying weights) is only impacted within the bandwidth of the kernel.

  • $\begingroup$ This clears things up greatly, this was what I was trying to remember! $\endgroup$ – Erosennin Jun 24 '17 at 9:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.