Does feature standardization always make sense?

I wonder if feature scaling like this makes always sense for neural networks:

Let $T$ be the training set and $x_i \in \mathbb{R}^n$ with $d_i \in T$ be the feature vector of $d_i$. Then add another preprocessing step so that $x_i' \gets \frac{x_i - \text{mean}(T)}{\max(T) - \min(T)}$ where $\max$ and $\min$ get applied seperately for each dimension.

This preprocessing step guarantees that for each feature you will get a mean of $0$ and a range of 1. I've heard that this is desired for neural nets. Do you know any sources for that? (Or sources that claim that feature normalization is not always good?)

Note: The range is1, not necessary the variance. The variance of a random variable $X$ is calculated like this:

$$Var(X) = E(X^2) - (\underbrace{E(X)}_{=0})^2 = E(X^2)$$.

If you have, for example, $X$ with $P(-0.5) = 0.5 = P(+0.5)$ you have a variance of $Var(X) = E(X^2) - E(X)^2 = (0.5 \cdot 0.25 + 0.5 \cdot 0.25) - 0 = 0.25$.

As $\max(X) - \min(X) = 0.5 - (-0.5) = 1$ and $\text{mean}(X) = 0$, feature standardization will not change anything

• (note its setting range to 1, not variance). duplicate of stats.stackexchange.com/questions/41704/… Aug 18, 2014 at 16:53
• @seanv507: Thanks for the range-note. I've fixed it and added an explanation. But my question is not a duplicate, as I am asking for reference. Aug 18, 2014 at 17:12
• Ignoring neural networks, if you're doing numerical work in floating point with limited precision, the best precision is near 0. The range [-1,1] has about 10 bits more precision than the range [999,1001]. Aug 26, 2014 at 12:31
• @MSalters: I don't think that is relevant. On the one hand, you can't get more precision than you had before. When your measurement has only two significant digits, there is no algorithm to get more. On the other hand, that would only be another reason to always apply feature standardization. Aug 26, 2014 at 13:44
• @moose: True for your measurements, not true for your computations (e.g. in iterative backpropagation). Also, remember that you're going to calculate (and minimize) an error which is the difference between your measurement and your network prediction. Aug 26, 2014 at 13:49

It depends a lot on your data.

Skewed features space make Gradient Descent way slower and sub-optimal (in general).

If they are heterogeneous kind of data, scaling may help. Let's say your features are Area (m^2), Temperature (°K) and so on. You have features of different size, so it may help convergence to make them of comparable size.

Moreover, in real applications, you may find useful to use dimensionality reduction with really high variance retain (say 0.99) to make your classifier more robust to noise and may help generalization. If heterogeneous data are processed with dimensionality reduction without scaling before it, meaningful data with low variance may be just dropped, if you do not properly scale before to apply it.

However, suppose that you are using color histogram features. You want to apply PCA at the 0.999 variance retain in order to avoid unpredictable behaviours of your classifier in real world deployment. If you perform features scaling before to apply PCA, you are not deleting anymore components with less variance, thus the one more affected by "noise". In this case I would not apply features scaling before dimensionality reduction. I would apply it after PCA, and before to feed features in the NN, always for easing convergence.

What about if you have histogram features and shape features? In this case, I would scale the two bunch of features separately. I would apply dimensionality reduction on them separately, and then I would scale them all together.

When dimensionality reduction is involved, you should apply features scaling always after its application, while you have to apply it before depending on the type of data.

Just for completeness, in few certain practical cases I encountered, not scaling features at all gave me slightly better results.

The best advice I can give you is to try.

From my experience, feature scaling is not always required. It's mainly to bring your features down to a comparable range so that the feature space is not skewed along any one particular feature. This will speed up convergence.

Whether you do it or not before training neural networks would depend on the data you're dealing with. For example, I won't do it when my feature vectors are binary one-hot or refer to indices of words (in language modelling). Likewise, also not if they are already in a comparable range, such as feature 1 in (0, 1) and feature 2 in (-1, 3) (arbitrary choice of ranges). On the other hand, if one of my features is in the range (0, 200) and another in the range (0, 1), I would definitely do it. You'll find a very nice summary on this subject from Prof. Andrew Ng in this video about feature scaling for gradient descent which is very pertinent to your question.

And this topic seems to have been already discussed on Cross-validated where the chosen answer states:

It's simply a case of getting all your data on the same scale: if the scales for different features are wildly different, this can have a knock-on effect on your ability to learn (depending on what methods you're using to do it). Ensuring standardised feature values implicitly weights all features equally in their representation.

You can find more about that question here:

How and why do normalization and feature scaling work?

• If I understand you correctly, there is no disadvantage in always applying feature standardization. Sometimes it might not be needed (I knew that before), but you didn't give an example where it is actively harmful. Aug 19, 2014 at 14:31
• I am yet to come across a case where it does something negative. For instance, I even tried applying feature standardization to binary one-hot vectors for which, ideally, you don't have to do anything and found that it didn't make anything worse. Just the learned weights were different because the data was presented differently. I also realized that this was pointless as my data was (after standardization) in a very similar range with the same regularities as one-hot vectors. Aug 20, 2014 at 14:02