8
$\begingroup$

I need to know why input normalization has no effect on polynomial regression. Here there is a good explanation proving that column normalization does not affect linear regression. But I need to know if it's the the same about Polynomial as well.

p.s. By normalizing I mean, for example, dividing the numbers by maximum in each column.

Thanks in advance,

$\endgroup$

1 Answer 1

5
$\begingroup$

Polynomial (least squares) regression of data $\newcommand{\y}{\mathrm{y}}\y$ (thought of as an $n$-vector $(y_i)$) against a variable $\newcommand{\x}{\mathrm{x}}\x$ (also an $n$-vector ($x_i)$) uses the variables $\mathrm {1} = \x^0$ (a constant $n$-vector), $\x = \x^1$, $\x^2 = (x_i^2)$, ... and $\x^d = (x_i^d)$. The fit $\hat \y$ is the projection of $\y$ onto the linear subspace $E$ spanned by these $d+1$ variables.

Suppose now that $\x$ is "normalized" by means of some affine transformation

$$\x^\prime = \alpha \x + \beta.$$

If we recompute the powers of the components $\x^\prime$ we find (after expanding them) that

$$x_i^{\prime k} = (\alpha x_i + \beta)^k = \sum_{j=0}^k c(k,j,\alpha,\beta)x_i^j$$

for some set of numbers $c(k,j,\alpha,\beta)$ (whose values we could write down explicitly, but the details do not matter). This exhibits every vector $(x_i^{\prime k})$ as a linear combination of the $x^j$, with $j$ varying from $0$ through nothing larger than $k$. Moreover, provided $\alpha \ne 0$ we can invert this process by noting

$$\x = \frac{1}{\alpha} \x^\prime - \frac{\beta}{\alpha}$$

and similarly expanding the powers of $\x$ in terms of powers of $\x^\prime$. Therefore the subspace spanned by $\x^\prime$ and its powers through degree $d$ is the same subspace $E$ as that spanned by $\x$ and its powers through degree $d$. Consequently the normalization does not change the geometric description, whence the fit with both models is identical, QED.


This argument generalizes, with essentially no change, to the cases where there may be more than one variable (expanded into a multivariate polynomial) as well as other non-polynomial covariates included in the model.

$\endgroup$
9
  • $\begingroup$ Thanks for your detailed answer. So, its kind of like proving induction. I am just not clear with the way of normalization in polynomial regression. Do we have to normalize all of the features x^2 x^3 and etc? or just normalizing x would be enough? Because if we normalize x then all other collumns would take effect. $\endgroup$ Jan 28, 2015 at 22:12
  • $\begingroup$ Exactly. The thread you refer to explains why separately normalizing the columns ($x, x^2, \ldots$) makes no difference, so the only issue left to address is whether the normalization of $x$ itself, followed by computation of its powers, makes any difference. However, the appropriate kind of "normalization" is standardization, yet standardization of $x$ does not generally standardize any of its powers. When these considerations become an issue, you really want to look into using orthogonal polynomials. $\endgroup$
    – whuber
    Jan 28, 2015 at 22:40
  • 1
    $\begingroup$ @Xiaoshi As far as I can see, he doesn't prove anything. His different fits in vastly different ranges of variables appear to result from floating point imprecision. Certainly scaling can affect the imprecision, but--as I prove here, mathematically--it has no effect on the model. $\endgroup$
    – whuber
    Jun 20, 2020 at 22:37
  • 1
    $\begingroup$ @Xiaoshi When scaling makes a difference in these linear models, you need to take precautions. Orthogonal polynomials and careful numerical algorithms (such as a Cholesky decomposition) come into play here. Search our site for these keywords ("orthogonal polynomial", "Cholesky", "floating point") for much more about these issues. $\endgroup$
    – whuber
    Jun 20, 2020 at 22:43
  • 1
    $\begingroup$ It is interesting to see that the practice can be very different from the theory, don't you think so? Thank you again for your explanation. $\endgroup$
    – John Smith
    Jun 20, 2020 at 22:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.