What makes linear regression with polynomial features curvy?

The following is my understanding of what happens: if I take a "two dimensional problem" e.g. I have $$X$$ as inputs and Y as the outcome and I add a feature $$x^2$$. This gives a problem an additional dimension and the linear fit on the $$x$$ and $$y$$ values define a line as well as the linear fit on $$x^2$$ and $$y$$ values and the two lines define a plane which is the best fit. Is this correct? How does this translate back to the 2 dimensional space? Does this somehow show up in two dimensions as curvy? How?

• $x^2$ is not an additional dimension because it's determined by $x$. the dimensions must be independent to some degree at least May 31 '19 at 19:57
• @Aksakal In the sense of dimensions of the column space of the model matrix, though, $x^2$ usually does introduce an additional dimension. That seems like a natural and useful way to understand this question.
– whuber
May 31 '19 at 20:15
• If we're thinking in terms of the design matrix $X$ that has observations as rows and variables as columns, then $x^2$ has its own column and in this regard adds a dimension. for instance, a covariance matrix $p\times p$ will one more dimension. moreover, in many cases the matix will even retain its rank $p$ despite $x^2$ being dependent on $x$, because it is not linearly dependent. that's why polynomial regression often works. however, it may sometimes fail due to collinearity or condition. May 31 '19 at 20:21
• I would suggest using orthogonal polynomials though. they're free of dependency issues of simple polynomials May 31 '19 at 20:22
• Using orthogonal polynomials instead of simpler ones doesn't change the result - that is, the estimated fit is the same -, although orthogonal polynomials have some practical advantages. That's not different from most multivariate regression problems, where predictors are correlated.
– Pere
Jun 1 '19 at 8:19

This is a piece of a plane in 3D.

Here is the same plane with coordinates shown and a set of points selected along its $$x$$ axis.

The third coordinate is used to plot the squares of these $$x$$ values, producing points along a parabola at the base of the coordinate box.

A vertical "curtain" through the parabola intersects the plane at all the points directly above the parabola. This intersection is a curve.

A polynomial model supposes the response $$y$$ (graphed in the vertical direction) differs from the height of this plane by random amounts. The values of $$y$$ corresponding to these $$x$$ coordinates are shown as red dots.

Consequently, the $$(x,y)$$ points lie along a curve--this projection--rather than a line, even though the model of the response is based on the plane originally shown.

Moral

When the explanatory variables clearly lie on a curve, the responses will appear to lie on a curve, too.

• Thank you so much, this was so helpful. Jun 2 '19 at 12:08

If you have a single independent variable x and a single dependent variable y, then "y = f(x)" is usually considered as two dimensional even if the relationship between these two variables is complicated. As a hypothetical example, if an experimental model is "pressure = a * temperature + b * log(temperature) - c * sine(temperature)" there are only two variables, temperature and pressure. For this reason, such a relationship could be plotted as a curved line on a flat plane.

If the model had two independent variables, such as "pressure = a * log(temperature) - b * exp(altitude)", this has the form of "z = f(x,y)" and could be plotted as a 3D surface.