I need to know why input normalization has no effect on polynomial regression. Here there is a good explanation proving that column normalization does not affect linear regression. But I need to know if it's the the same about Polynomial as well.

p.s. By normalizing I mean, for example, dividing the numbers by maximum in each column.

Thanks in advance,


Polynomial (least squares) regression of data $\newcommand{\y}{\mathrm{y}}\y$ (thought of as an $n$-vector $(y_i)$) against a variable $\newcommand{\x}{\mathrm{x}}\x$ (also an $n$-vector ($x_i)$) uses the variables $\mathrm {1} = \x^0$ (a constant $n$-vector), $\x = \x^1$, $\x^2 = (x_i^2)$, ... and $\x^d = (x_i^d)$. The fit $\hat \y$ is the projection of $\y$ onto the linear subspace $E$ spanned by these $d+1$ variables.

Suppose now that $\x$ is "normalized" by means of some affine transformation

$$\x^\prime = \alpha \x + \beta.$$

If we recompute the powers of the components $\x^\prime$ we find (after expanding them) that

$$x_i^{\prime k} = (\alpha x_i + \beta)^k = \sum_{j=0}^k c(k,j,\alpha,\beta)x_i^j$$

for some set of numbers $c(k,j,\alpha,\beta)$ (whose values we could write down explicitly, but the details do not matter). This exhibits every vector $(x_i^{\prime k})$ as a linear combination of the $x^j$, with $j$ varying from $0$ through nothing larger than $k$. Moreover, provided $\alpha \ne 0$ we can invert this process by noting

$$\x = \frac{1}{\alpha} \x^\prime - \frac{\beta}{\alpha}$$

and similarly expanding the powers of $\x$ in terms of powers of $\x^\prime$. Therefore the subspace spanned by $\x^\prime$ and its powers through degree $d$ is the same subspace $E$ as that spanned by $\x$ and its powers through degree $d$. Consequently the normalization does not change the geometric description, whence the fit with both models is identical, QED.

This argument generalizes, with essentially no change, to the cases where there may be more than one variable (expanded into a multivariate polynomial) as well as other non-polynomial covariates included in the model.

| cite | improve this answer | |
  • $\begingroup$ Thanks for your detailed answer. So, its kind of like proving induction. I am just not clear with the way of normalization in polynomial regression. Do we have to normalize all of the features x^2 x^3 and etc? or just normalizing x would be enough? Because if we normalize x then all other collumns would take effect. $\endgroup$ – Mohammadreza Jan 28 '15 at 22:12
  • $\begingroup$ Exactly. The thread you refer to explains why separately normalizing the columns ($\x, \x^2, \ldots$) makes no difference, so the only issue left to address is whether the normalization of $\x$ itself, followed by computation of its powers, makes any difference. However, the appropriate kind of "normalization" is standardization, yet standardization of $\x$ does not generally standardize any of its powers. When these considerations become an issue, you really want to look into using orthogonal polynomials. $\endgroup$ – whuber Jan 28 '15 at 22:40
  • $\begingroup$ If we normalize column x by just dividing by the magnitude of maximum, Then I think its power would be also normalized. Am I right or there is a misunderstanding? Thanks whuber. $\endgroup$ – Mohammadreza Jan 29 '15 at 19:35
  • $\begingroup$ That would be correct only when the maximum of $\x$ is strictly positive and the minimum of $\x$ is no greater in size than the maximum. For instance, when $\x=(2,-4)$ your normalized values would be $(1,-2)$. Their squares are $(1,4)$, which is not normalized to have a maximum of $1$. $\endgroup$ – whuber Jan 29 '15 at 20:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.