I know sometimes it is necessary to normalize/standardize the predictors for supervised learning but is it fair to say that you can always do this without hurting performance? Or are there scenarios when using the raw data can be better?

  • 1
    $\begingroup$ In ridge regression and other forms of regularisation based on coefficient norms you will get different answers. If the original scale is meaningful then the raw data may be better (eg signal processing, 1 input signal other input noise of lower variance) regularisation enforces small inputs should have small effects. $\endgroup$ – seanv507 Sep 29 '16 at 3:16

Typically standardization is done for:

  • easing interpretation
    • estimated coefficients etc... are relative to standard deviations of the underlying variable
  • numerical issues
    • Having very big numbers and very small numbers in your data matrix $X$ can lead to numerical problems. Eg. the matrix $X'X$ may have an extremely high condition number. You'll find polynomial curve fitting with ordinary least squares works better when you standardize first in the sense you can include high order polynomial terms without getting an ill-conditioned linear system.
    • Various other problems in optimization may come up for other machine learning techniques if you have poorly scaled data.

That is, it's generally not about getting a better fit in a theoretical sense. And a big category where standardization theoretically doesn't matter at all for predictive power is unconstrained linear models. (The big caveat is the numerical, computational issues mentioned above.)

Of course the general answer is it depends. If you take a non-linear transformation of your data (eg. exp), it almost certainly will matter whether it has first been standardized. Which is better then? I don't know a general way to answer that. In macro-finance, you often work with the logarithm of variables (eg. log GDP, log price, differences in log price etc...). Obviously, standardizing before you take the log would be non-sensical / insane.

Standardization and linear models

First observe the following fact:

Standardization is a linear transformation if you include a constant in your data

Let $\mu$ be the standard deviation of your random variable $x_1$. Let $\sigma$ be the standard deviation.

Observe that standardizing your variable is an affine transformation of $x_1$.

$$ f(x_1) = \frac{x_1 - \mu}{\sigma} $$

When you include a constant in your regression, you're effectively creating a new variable $x_0 = 1$. Observe that $g$ is linear function of $x_0$ and $x_1$.

$$g(x_0, x_1) = \frac{x_1 - x_0 \mu}{\sigma} $$

Including a variable which is always 1 is a ubiquitous trick to model affine transformations as linear transformations.

Example with data matrices

If you have the data matrix $X$ where the first column is a column of ones, and the second column is a vector $\mathbf{x}$ denoting all the observations of your random variable $\hat{x}$. Something like:

$$ X = \left[ \begin{array}{cc} 1 & 2.5 \\ 1 & -1.3 \\ 1 & 13.2 \end{array} \right] $$

If you're simply standardizing $x$ (i.e. the data represented in your second column) you can represent this as a linear transformation $A$ defined by the matrix:

$$ A = \left[ \begin{array}{ccc} 1 & - \frac{\mu_x}{\sigma_x} \\ 0 & \frac{1}{\sigma_x} \end{array} \right] $$

Multiplying $X$ by $A$ (i.e. computing $XA$) is basically equivalent to subtracting the mean for each non-constant column and dividing by the standard deviation.

Linear models will typically be invariant to a full-rank linear transformation of the underlying data

Example: equivalency of the least squares estimator

The least squares estimator is: $$ b = (X'X)^{-1} X'y $$

The OLS estimator on the transformed data $XA$ is: $$ \begin{align*} \hat{b} &= (A'X'XA)^{-1} A'X'y \\ &= A^{-1} (X'X)^{-1} A'^{-1}A'X'y \\ &= A^{-1} b \end{align*} $$

Your estimated coefficients are different, but the predictions are the same: $Xb = (XA)\left(A^{-1}b \right) = \hat{X}\hat{b} $.

Cases where it's not equivalent

If you do something non-linear with your data, then all bets are off. If I regress $y$ on a constant and $\exp(x)$ this won't be equivalent to regressing $y$ on a constant and $\exp(\hat{x})$ where $\hat{x} = \frac{x - \mu}{\sigma}$.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.