Is it a good practice to always scale/normalize data for machine learning?

My understanding is that when some features have different ranges in their values (for example, imagine one feature being the age of a person and another one being their salary in USD) will affect negatively algorithms because the feature with bigger values will take more influence, is it a good practice to simply ALWAYS scale/normalize the data?

It looks to me that if the values are already similar among then, then normalizing them will have little effect, but if the values are very different normalization will help, however it feels too simple to be true :)

Am I missing something? Are there situations/algorithms were actually it is desirable to let some features to deliberately outweigh others?

-
– Henry Jan 7 at 15:17

First things first, I don't think there are many questions of the form "Is it a good practice to always X in machine learning" where the answer is going to be definitive. Always? Always always? Across parametric, non-parametric, Bayesian, Monte Carlo, social science, purely mathematic, and million feature models? That'd be nice, wouldn't it!

Concretely though, here are a few ways in which: it just depends.

Some times when normalizing is good:

1) Several algorithms, in particular SVMs come to mind, can sometimes converge far faster on normalized data (although why, precisely, I can't recall).

2) When your model is sensitive to magnitude, and the units of two different features are different, and arbitrary. This is like the case you suggest, in which something gets more influence than it should.

But of course -- not all algorithms are sensitive to magnitude in the way you suggest. Linear regression coefficients will be identical if you do, or don't, scale your data, because it's looking at proportional relationships between them.

Some times when normalizing is bad:

1) When you want to interpret your coefficients, and they don't normalize well. Regression on something like dollars gives you a meaningful outcome. Regression on proportion-of-maximum-dollars-in-sample might not.

2) When, in fact, the units on your features are meaningful, and distance does make a difference! Back to SVMs -- if you're trying to find a max-margin classifier, then the units that go into that 'max' matter. Scaling features for clustering algorithms can substantially change the outcome. Imagine four clusters around the origin, each one in a different quadrant, all nicely scaled. Now, imagine the y-axis being stretched to ten times the length of the the x-axis. instead of four little quadrant-clusters, you're going to get the long squashed baguette of data chopped into four pieces along its length! (And, the important part is, you might prefer either of these!)

In I'm sure unsatisfying summary, the most general answer is that you need to ask yourself seriously what makes sense with the data, and model, you're using.

-
Thanks for the answer, but here goes another question, you say that in regression models normalizing for example salary (1000-100000) and (say) age (10-80) will not help a lot (specially because one looses the meaning of the numbers...), however, if I do not normalize that, it will happen that the salary will outweigh the age, wouldn't it? – Juan Antonio Gomez Moriano Jan 7 at 21:35
What do you mean by outweigh? I'm assuming salary and age are both independent variables here. Normalizing them only changes the units on their coefficients. But for example's sake, because it's clearer than a normalizing constant, lets divide by say 1000. But then your coefficient just means change-per-\$1000-change, as opposed to change-per-\$1-change. The numbers are different, but you should always be thinking about what your coefficients mean -- you just can't stop with the numbers alone. – Sophologist Jan 7 at 21:48
By normalizing them i mean to use a function like scale in r, such as data$age<-scale(data$age) and data$salary <- scale(data$salary). At the end of the day, when using something like logistic regression, one is just learning the parameters for a vector, correct? if such vector contains some variables whose values are in a much bigger range than others wouldn't that be a problem? I have been playing with and without scaled data for logistic regression and scaling it seems to help... Am I missing something? – Juan Antonio Gomez Moriano Jan 7 at 21:51
Normalizing, I understand -- you used the word outweigh, and I still don't understand how you're using it. And again, scaling "seems to help" -- what do you mean? Help how? – Sophologist Jan 7 at 21:57
Possibly it is my English :). What I mean is that given that one variable (salary) has a wider range than other (say age), will mean that the age will have very little importance when it comes to perform predictions while salary will be the most determinant factor and that is not always desirable. By "seems to help" I mean that when trying a model trained with/without scaled values, the scaled one generalizes better (using the cross validation set). I hope this clarifies :) – Juan Antonio Gomez Moriano Jan 7 at 22:06

Well I believe a more geometric point of view will help better decide whether normalization helps or not. Imagine your problem of interest has only two features and they range different. Then geometrically, the data points are spread around and form an ellipsoid. However, if the features are normalized they will be more concentrated and hopefully form a unit circle and make the covariance diagonal or at least close to diagonal. This is what the idea is behind methods such as batch-normalizing the intermediate representations of data in neural networks. Using BN the convergence speed increases amazingly (maybe 5-10 times) since the gradient can easily help the gradients do what they are supposed to do in order to reduce the error.

In the unnormalized case, gradient-based optimization algorithms will have a very hard time to move the weight vectors towards a good solution. However, the cost surface for the normalized case is less elongated and gradient-based optimization methods will do much better and diverge less.

This is certainly the case for linear models and specially the ones whos cost function is a measure of divergence of the model's output and the target (e.g. linear regression with MSE cost function), but might not be necessarily the case in the non-linear ones. However, normalization does not hurt the for nonlinear models but not doing it for linear models will do hurt.

The picture below could be [roughly] viewed as example of an elongated error surface in which the gradient-based methods could have hard time to help the weight vectors move towards the local optima.

-
However, normalization does not hurt the for nonlinear models but not doing it for linear models will do hurt. I find this sentence hard to understand. Is it (roughly spoken) irrelevant for non-linear models whether the data is normalized? Not doing it for linear models will break something, but can you specify better for non-linear models? Maybe try to avoid mixing positive and negative expressions in this one sentence. – J_F_B_M Jan 7 at 11:28
In linear models features wirh large ranges will induce high variance to the model and therefore may become unnecessarily important. For instance in PCA those features will hqve much larger eigenvalues than others. However in nonlinear models (depends on many factors) this might not be the case since the model may completely change the representation of the data through the nonlinearities. So it's not as easy to analyze what goes on in a nonlinear model and how unnormalized features affect the output. – Amir Jan 7 at 11:36
So basically the effect of normalization in non-linear models is hard to predict and should be decided on a case-by-case basis? – J_F_B_M Jan 7 at 11:39
True. Although it's been oroven empirically that normalization helps in nonlinear models as well. – Amir Jan 7 at 11:41
Thank you for this clarification. This helped me to understand your answer better. – J_F_B_M Jan 7 at 12:12

Let me tell you the story of how I learned the importance of normalization.

I was trying to classify a handwritten digits data (it is a simple task of classifying features extracted from images of hand-written digits) with Neural Networks as an assignment for a Machine Learning course.
Just like anyone else, I started with a Neural Network library/tool, fed it with the data and started playing with the parameters. I tried changing number of layers, the number of neurons and various activation functions. None of them yielded expected results (accuracy around 0.9).

The Culprit? The scaling factor (s) in the activation function = $\frac{s}{1+e^{-s.x}}$-1. If the parameter s is not set, the activation function will either activate every input or nullify every input in every iteration. Which obviously led to unexpected values for model parameters. My point is, it is not easy to set s when the input x is varying over large values.

As some of the other answers have already pointed it out, the "good practice" as to whether to normalize the data or not depends on the data, model, and application. By normalizing, you are actually throwing away some information about the data such as the absolute maximum and minimum values. So, there is no rule of thumb.

Thanks.

-

As others said, normalization is not always applicable; e.g. from a practical point of view.

In order to be able to scale or normalize features to a common range like [0,1], you need to know the min/max (or mean/stdev depending on which scaling method you apply) of each feature. IOW: you need to have all the data for all features before you start training.

Many practical learning problems don't provide you with all the data a-priori, so you simply can't normalize. Such problems require an online learning approach.

However, note that some online (as opposed to batch learning) algorithms which learn from one example at a time, support an approximation to scaling/normalization. They learn the scales and compensate for them, iteratively. vowpal wabbit for example iteratively normalizes for scale by default (unless you explicitly disable auto-scaling by forcing a certain optimization algorithm like naive --sgd)

-

I was trying to solve ridge regression problem using gradient descent. Now without normalization I set some appropriate step size and ran the code. In order to make sure my coding was error-free, I coded the same objective in CVX too. Now CVX took only few iterations to converge to a certain optimal value but I ran my code for the best step size I could find by 10k iterations and I was close to the optimal value of CVX but still not exact.

After normalizing the data-set and feeding it to my code and CVX, I was surprised to see that now convergence only took 100 iterations and the optimal value to which gradient descent converged was exactly equal to that of CVX.

Also the amount of "explained variance" by model after normalization was more compared to the original one. So just from this naive experiment I realized that as far as regression problem is concerned I would go for normalization of data. BTW here normalization implies subtracting by mean and dividing by standard deviation.

For backing me on regression please see this relevant question and discussion on it:
When conducting multiple regression, when should you center your predictor variables & when should you standardize them?

-