The following is my understanding of what happens: if I take a "two dimensional problem" e.g. I have $X$ as inputs and Y as the outcome and I add a feature $x^2$. This gives a problem an additional dimension and the linear fit on the $x$ and $y$ values define a line as well as the linear fit on $x^2$ and $y$ values and the two lines define a plane which is the best fit. Is this correct? How does this translate back to the 2 dimensional space? Does this somehow show up in two dimensions as curvy? How?
$\begingroup$
$\endgroup$
5
-
1$\begingroup$ $x^2$ is not an additional dimension because it's determined by $x$. the dimensions must be independent to some degree at least $\endgroup$– AksakalMay 31, 2019 at 19:57
-
4$\begingroup$ @Aksakal In the sense of dimensions of the column space of the model matrix, though, $x^2$ usually does introduce an additional dimension. That seems like a natural and useful way to understand this question. $\endgroup$– whuber ♦May 31, 2019 at 20:15
-
$\begingroup$ If we're thinking in terms of the design matrix $X$ that has observations as rows and variables as columns, then $x^2$ has its own column and in this regard adds a dimension. for instance, a covariance matrix $p\times p$ will one more dimension. moreover, in many cases the matix will even retain its rank $p$ despite $x^2$ being dependent on $x$, because it is not linearly dependent. that's why polynomial regression often works. however, it may sometimes fail due to collinearity or condition. $\endgroup$– AksakalMay 31, 2019 at 20:21
-
$\begingroup$ I would suggest using orthogonal polynomials though. they're free of dependency issues of simple polynomials $\endgroup$– AksakalMay 31, 2019 at 20:22
-
2$\begingroup$ Using orthogonal polynomials instead of simpler ones doesn't change the result - that is, the estimated fit is the same -, although orthogonal polynomials have some practical advantages. That's not different from most multivariate regression problems, where predictors are correlated. $\endgroup$– PereJun 1, 2019 at 8:19
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
This is a piece of a plane in 3D.
Here is the same plane with coordinates shown and a set of points selected along its $x$ axis.
The third coordinate is used to plot the squares of these $x$ values, producing points along a parabola at the base of the coordinate box.
A vertical "curtain" through the parabola intersects the plane at all the points directly above the parabola. This intersection is a curve.
A polynomial model supposes the response $y$ (graphed in the vertical direction) differs from the height of this plane by random amounts. The values of $y$ corresponding to these $x$ coordinates are shown as red dots.
Consequently, the $(x,y)$ points lie along a curve--this projection--rather than a line, even though the model of the response is based on the plane originally shown.
Moral
When the explanatory variables clearly lie on a curve, the responses will appear to lie on a curve, too.
-
1
$\begingroup$
$\endgroup$
0
If you have a single independent variable x and a single dependent variable y, then "y = f(x)" is usually considered as two dimensional even if the relationship between these two variables is complicated. As a hypothetical example, if an experimental model is "pressure = a * temperature + b * log(temperature) - c * sine(temperature)" there are only two variables, temperature and pressure. For this reason, such a relationship could be plotted as a curved line on a flat plane.
If the model had two independent variables, such as "pressure = a * log(temperature) - b * exp(altitude)", this has the form of "z = f(x,y)" and could be plotted as a 3D surface.
answered May 31, 2019 at 19:50