2
$\begingroup$

I understand that, ostensibly, polynomial regression is useful for extending the linearity assumption in least squares regression. It can model nonlinear relationships between a predictor and response.

But, it seems that it only works with a model limited to a single predictor.

For an example, I'm working with the Boston housing dataset. It has information on median income in 500 parcels throughout Boston, described by thirteen variables.

It is unreasonable to expect the relationship can be summarized by just one of those variables. I suppose that makes it unreasonable to also expect it to be summarized by a polynomial regression with just one of those variables?

Am I wrong? Is my assumption that a polynomial regression is largely limited to one predictor off-base?

$\endgroup$
  • 2
    $\begingroup$ Yes you're wrong $\endgroup$ – Jake Westfall Jan 16 at 19:48
  • $\begingroup$ Great, tell me more! $\endgroup$ – Sebastian Jan 16 at 19:53
  • $\begingroup$ Related- stats.stackexchange.com/questions/387477/… $\endgroup$ – Sebastian Jan 16 at 19:53
  • $\begingroup$ Why do you think that it is largely limited to one predictor? Watch a Youtube video on splines $\endgroup$ – peteR Jan 16 at 20:13
  • 2
    $\begingroup$ Search for "multiple linear regression". $\endgroup$ – James Phillips Jan 16 at 21:11
4
$\begingroup$

You can run multiple linear regression, where you relate an outcome to multiple predictors. For instance, you could have two predictors:

$$ y = \beta_0+\beta_1x_1 + \beta_2x_2 + \epsilon. $$

To model nonlinearities in the predictors, you can simply polynomially transform either or both predictors separately, e.g.:

$$ y = \beta_0+\beta_{11}x_1 + \beta_{12}x_1^2 +\beta_{21}x_2 + \beta_{22}x_2^2 + \beta_{23}x_2^3+ \epsilon. $$

Or you can include interaction terms, like:

$$ y = \beta_0+\beta_{11}x_1 + \beta_{12}x_1^2 +\beta_{21}x_2 + \beta_{22}x_2^2 + \beta_{23}x_2^3 + \gamma_{11}x_1x_2 + \gamma_{12}x_1x_2^2 + \epsilon. $$

Splines are pretty much always better than straight polynomials, just like in the single predictor case.

And yes, it can be hard to determine how many knots your splines should have, just like in the single predictor case.

$\endgroup$
4
$\begingroup$

First of all, as other have pointed out, polynomial regression can be applied in either univariate or multivariate situations --- it is not limited to univariate regression. Polynomial functions can operate on several variables, and this includes polynomial terms in each of those variables, plus interaction terms.

As to usefulness, the main reason that polynomial regression is such a useful model form is that it is actually a type of linear regression (i.e., it is linear with respect to the coefficients in the model) but it can be used to give an arbitrarily close approximation to a wide variety of non-linear functions (i.e., notlinear with respect to the explanatory variables, but still linear with respect to the coefficients). This aspect of the model comes from the method of Taylor approximation. For a wide class of non-linear functions $f$ taken over our limited data range, you can approximate the function up to an arbitrary level of accuracy by the polynomial:

$$f_{(k)}(x|a) = \sum_{i=1}^k \frac{1}{k!} (x-a)^k \frac{d^kf}{dx^k}(a),$$

where $a \in \mathbb{R}$ is a chosen base point. (Note that the Taylor approximation can be extended to multiple variables, and so it can be used to yield a multivariate polynomial regression.) Taylor polynomials are polynomial approximations to non-linear functions, so they form the justification for why polynomial regression is regarded as a standard model form that can model a broad class of relationships reasonably well.

Now, it is important to note that in polynomial regression we are limited in the degree of the polynomial we can use, and the number of interaction terms between different explanatory variables, by the fact that we have a limited data set and we don't want to lose too many degrees-of-freedom. If we have a large number of data points relative to the number of variables then it is possible to fit polynomial regression to a reasonably high degree, and this will generally give you a good approximation to a wide class of functions that are non-linear with respect to the explanatory variables. (A notable exception to this is periodic functions, which are better approximated by Fourier approximation, which gives you periodic regression.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.