Is it a good practice to always scale/normalize data for machine learning? [duplicate]

Question

My understanding is that when some features have different ranges in their values (for example, imagine one feature being the age of a person and another one being their salary in USD) will affect negatively algorithms because the feature with bigger values will take more influence, is it a good practice to simply ALWAYS scale/normalize the data?

It looks to me that if the values are already similar among then, then normalizing them will have little effect, but if the values are very different normalization will help, however it feels too simple to be true :)

Am I missing something? Are there situations/algorithms were actually it is desirable to let some features to deliberately outweigh others?

Answer 1

First things first, I don't think there are many questions of the form "Is it a good practice to always X in machine learning" where the answer is going to be definitive. Always? Always always? Across parametric, non-parametric, Bayesian, Monte Carlo, social science, purely mathematic, and million feature models? That'd be nice, wouldn't it!

Concretely though, here are a few ways in which: it just depends.

Some times when normalizing is good:

1) Several algorithms, in particular SVMs come to mind, can sometimes converge far faster on normalized data (although why, precisely, I can't recall).

2) When your model is sensitive to magnitude, and the units of two different features are different, and arbitrary. This is like the case you suggest, in which something gets more influence than it should.

But of course -- not all algorithms are sensitive to magnitude in the way you suggest. Linear regression coefficients will be identical if you do, or don't, scale your data, because it's looking at proportional relationships between them.

Some times when normalizing is bad:

1) When you want to interpret your coefficients, and they don't normalize well. Regression on something like dollars gives you a meaningful outcome. Regression on proportion-of-maximum-dollars-in-sample might not.

2) When, in fact, the units on your features are meaningful, and distance does make a difference! Back to SVMs -- if you're trying to find a max-margin classifier, then the units that go into that 'max' matter. Scaling features for clustering algorithms can substantially change the outcome. Imagine four clusters around the origin, each one in a different quadrant, all nicely scaled. Now, imagine the y-axis being stretched to ten times the length of the the x-axis. instead of four little quadrant-clusters, you're going to get the long squashed baguette of data chopped into four pieces along its length! (And, the important part is, you might prefer either of these!)

In I'm sure unsatisfying summary, the most general answer is that you need to ask yourself seriously what makes sense with the data, and model, you're using.

Answer 2

Well, I believe a more geometric point of view will help better decide whether normalization helps or not. Imagine your problem of interest has only two features and they range differently. Then geometrically, the data points are spread around and form an ellipsoid. However, if the features are normalized they will be more concentrated and hopefully, form a unit circle and make the covariance diagonal or at least close to diagonal. This is what the idea is behind methods such as batch-normalizing the intermediate representations of data in neural networks. Using BN the convergence speed increases amazingly (maybe 5-10 times) since the gradient can easily help the gradients do what they are supposed to do in order to reduce the error.

In the unnormalized case, gradient-based optimization algorithms will have a very hard time to move the weight vectors towards a good solution. However, the cost surface for the normalized case is less elongated and gradient-based optimization methods will do much better and diverge less.

This is certainly the case for linear models and especially the ones whose cost function is a measure of divergence of the model's output and the target (e.g. linear regression with MSE cost function), but might not be necessarily the case in the non-linear ones. Normalization does not hurt for the nonlinear models; not doing it for linear models will hurt.

The picture below could be [roughly] viewed as the example of an elongated error surface in which the gradient-based methods could have a hard time to help the weight vectors move towards the local optima.

Answer 3

Let me tell you the story of how I learned the importance of normalization.

I was trying to classify a handwritten digits data (it is a simple task of classifying features extracted from images of hand-written digits) with Neural Networks as an assignment for a Machine Learning course.
Just like anyone else, I started with a Neural Network library/tool, fed it with the data and started playing with the parameters. I tried changing number of layers, the number of neurons and various activation functions. None of them yielded expected results (accuracy around 0.9).

The Culprit? The scaling factor (s) in the activation function = $\frac{s}{1+e^{-s.x}}$-1. If the parameter s is not set, the activation function will either activate every input or nullify every input in every iteration. Which obviously led to unexpected values for model parameters. My point is, it is not easy to set s when the input x is varying over large values.

As some of the other answers have already pointed it out, the "good practice" as to whether to normalize the data or not depends on the data, model, and application. By normalizing, you are actually throwing away some information about the data such as the absolute maximum and minimum values. So, there is no rule of thumb.

Answer 4

As others said, normalization is not always applicable; e.g. from a practical point of view.

In order to be able to scale or normalize features to a common range like [0,1], you need to know the min/max (or mean/stdev depending on which scaling method you apply) of each feature. IOW: you need to have all the data for all features before you start training.

Many practical learning problems don't provide you with all the data a-priori, so you simply can't normalize. Such problems require an online learning approach.

However, note that some online (as opposed to batch learning) algorithms which learn from one example at a time, support an approximation to scaling/normalization. They learn the scales and compensate for them, iteratively. vowpal wabbit for example iteratively normalizes for scale by default (unless you explicitly disable auto-scaling by forcing a certain optimization algorithm like naive --sgd)

Answer 5

Scaling/normalizing does change your model slightly. Most of the time this corresponds to applying an affine function. So you have $Z=A_X+B_XXC_X$ where $X$ is your "input/original data" (one row for each training example, one column for each feature). Then $A_X,B_X,C_X$ are matrices that are typically functions of $X$. The matrix $Z$ is what you feed into your ML algorithm.

Now, suppose you want to predict for some new sample. But you only have $X_{new}$ and not $Z_{new}$. You should be applying the function $Z_{new}=A_X+B_XX_{new}C_X$. That is, use the same $A_X,B_X,C_X$ from the training dataset, rather that re-estimate them. This makes these matrices have the same form as other parameters in your model.

While they are often equivalent in terms of the predicted values you get from the training dataset, it certainly isn't on new data for predictions. A simple example, predict for $1$ new observation, standardising this (subtract mean, divide by sd) will always return zero.

Answer 6

For machine learning models that include coefficients (e.g. regression, logistic regression, etc) the main reason to normalize is numerical stability. Mathematically, if one of your predictor columns is multiplied by 10^6, then the corresponding regression coefficient will get multiplied by 10^{-6} and the results will be the same.

Computationally, your predictors are often transformed by the learning algorithm (e.g. the matrix X of predictors in a regression becomes X'X) and some of those transformations can result in lost numerical precision if X is very large or very small. If your predictors are on the scale of 100's then this won't matter. If you're modeling grains of sand, astronomical units, or search query counts then it might.

Answer 7

I was trying to solve ridge regression problem using gradient descent. Now without normalization I set some appropriate step size and ran the code. In order to make sure my coding was error-free, I coded the same objective in CVX too. Now CVX took only few iterations to converge to a certain optimal value but I ran my code for the best step size I could find by 10k iterations and I was close to the optimal value of CVX but still not exact.

After normalizing the data-set and feeding it to my code and CVX, I was surprised to see that now convergence only took 100 iterations and the optimal value to which gradient descent converged was exactly equal to that of CVX.

Also the amount of "explained variance" by model after normalization was more compared to the original one. So just from this naive experiment I realized that as far as regression problem is concerned I would go for normalization of data. BTW here normalization implies subtracting by mean and dividing by standard deviation.

For backing me on regression please see this relevant question and discussion on it:
When conducting multiple regression, when should you center your predictor variables & when should you standardize them?