7
$\begingroup$

The following is my understanding of what happens: if I take a "two dimensional problem" e.g. I have $X$ as inputs and Y as the outcome and I add a feature $x^2$. This gives a problem an additional dimension and the linear fit on the $x$ and $y$ values define a line as well as the linear fit on $x^2$ and $y$ values and the two lines define a plane which is the best fit. Is this correct? How does this translate back to the 2 dimensional space? Does this somehow show up in two dimensions as curvy? How?

$\endgroup$
  • 1
    $\begingroup$ $x^2$ is not an additional dimension because it's determined by $x$. the dimensions must be independent to some degree at least $\endgroup$ – Aksakal May 31 at 19:57
  • 4
    $\begingroup$ @Aksakal In the sense of dimensions of the column space of the model matrix, though, $x^2$ usually does introduce an additional dimension. That seems like a natural and useful way to understand this question. $\endgroup$ – whuber May 31 at 20:15
  • $\begingroup$ If we're thinking in terms of the design matrix $X$ that has observations as rows and variables as columns, then $x^2$ has its own column and in this regard adds a dimension. for instance, a covariance matrix $p\times p$ will one more dimension. moreover, in many cases the matix will even retain its rank $p$ despite $x^2$ being dependent on $x$, because it is not linearly dependent. that's why polynomial regression often works. however, it may sometimes fail due to collinearity or condition. $\endgroup$ – Aksakal May 31 at 20:21
  • $\begingroup$ I would suggest using orthogonal polynomials though. they're free of dependency issues of simple polynomials $\endgroup$ – Aksakal May 31 at 20:22
  • 2
    $\begingroup$ Using orthogonal polynomials instead of simpler ones doesn't change the result - that is, the estimated fit is the same -, although orthogonal polynomials have some practical advantages. That's not different from most multivariate regression problems, where predictors are correlated. $\endgroup$ – Pere Jun 1 at 8:19
15
$\begingroup$

This is a piece of a plane in 3D.

Figure 1

Here is the same plane with coordinates shown and a set of points selected along its $x$ axis.

Figure 2

The third coordinate is used to plot the squares of these $x$ values, producing points along a parabola at the base of the coordinate box.

Figure 3

A vertical "curtain" through the parabola intersects the plane at all the points directly above the parabola. This intersection is a curve.

Figure 4

A polynomial model supposes the response $y$ (graphed in the vertical direction) differs from the height of this plane by random amounts. The values of $y$ corresponding to these $x$ coordinates are shown as red dots.

Figure 5

Consequently, the $(x,y)$ points lie along a curve--this projection--rather than a line, even though the model of the response is based on the plane originally shown.

Figure 6

Moral

When the explanatory variables clearly lie on a curve, the responses will appear to lie on a curve, too.

$\endgroup$
  • 1
    $\begingroup$ Thank you so much, this was so helpful. $\endgroup$ – user412953 Jun 2 at 12:08
3
$\begingroup$

If you have a single independent variable x and a single dependent variable y, then "y = f(x)" is usually considered as two dimensional even if the relationship between these two variables is complicated. As a hypothetical example, if an experimental model is "pressure = a * temperature + b * log(temperature) - c * sine(temperature)" there are only two variables, temperature and pressure. For this reason, such a relationship could be plotted as a curved line on a flat plane.

If the model had two independent variables, such as "pressure = a * log(temperature) - b * exp(altitude)", this has the form of "z = f(x,y)" and could be plotted as a 3D surface.

$\endgroup$
  • 1
    $\begingroup$ Ah, I think I got it, thanks a lot! $\endgroup$ – user412953 May 31 at 20:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.