Fitting polynomial in regression: undesirable nonlocal effects

Question

So we use polynomial regression for estimating conditional mean functions of unknown form, and especially higher order polynomials are flexible. I've learned in the past that especially higher-order polynomials have undesirable effects when being fitted: values of the response variable for some special segments of the predictor will affect badly estimated values for other segments of the predictor. I would be grateful for any further explanation on this matter!

Answer 1

Think about how polynomials behave. Graph any polynomial. Look at what happens when you extrapolate beyond the data. The function will go to negative or positive infinity. That is almost always undesirable. But if you only need to estimate in the range where you already have data, then it should be OK, usually.

Answer 2

Polynomials are notorious for undesirable non-local effects and aside from very low degree polynomials (2 or 3 in particular situations) are usually avoided for that reason. Peter's discussion gives some sense of why it happens, but in any case it's easy to observe in practice -- a simple case is to fit a high order polynomial to a few data points (e.g. a degree 6 polynomial to 9 data points say) and then notice the way the behavior of the whole fit alters as you change one of the points.

Splines are often used when there's no specific functional relationship in mind. For example, a similar situation when fitting natural cubic regression splines will have local effects - the fit for nearby points will change - but as you move more than a few knots away from the changing value, the fit doesn't alter. You can also parameterize your spline fit (via B-splines say) so the coefficients of the components are only affected locally.

Similarly, local regression methods (such as local linear regression via a bounded kernel supplying weights) is only impacted within the bandwidth of the kernel.