Does feature standardization always make sense?

Question

I wonder if feature scaling like this makes always sense for neural networks:

Let $T$ be the training set and $x_i \in \mathbb{R}^n$ with $d_i \in T$ be the feature vector of $d_i$. Then add another preprocessing step so that $x_i' \gets \frac{x_i - \text{mean}(T)}{\max(T) - \min(T)}$ where $\max$ and $\min$ get applied seperately for each dimension.

This preprocessing step guarantees that for each feature you will get a mean of $0$ and a range of 1. I've heard that this is desired for neural nets. Do you know any sources for that? (Or sources that claim that feature normalization is not always good?)

Note: The range is1, not necessary the variance. The variance of a random variable $X$ is calculated like this:

$$Var(X) = E(X^2) - (\underbrace{E(X)}_{=0})^2 = E(X^2)$$.

If you have, for example, $X$ with $P(-0.5) = 0.5 = P(+0.5)$ you have a variance of $Var(X) = E(X^2) - E(X)^2 = (0.5 \cdot 0.25 + 0.5 \cdot 0.25) - 0 = 0.25$.

As $\max(X) - \min(X) = 0.5 - (-0.5) = 1$ and $\text{mean}(X) = 0$, feature standardization will not change anything

Answer 1

It depends a lot on your data.

Skewed features space make Gradient Descent way slower and sub-optimal (in general).

If they are heterogeneous kind of data, scaling may help. Let's say your features are Area (m^2), Temperature (°K) and so on. You have features of different size, so it may help convergence to make them of comparable size.

Moreover, in real applications, you may find useful to use dimensionality reduction with really high variance retain (say 0.99) to make your classifier more robust to noise and may help generalization. If heterogeneous data are processed with dimensionality reduction without scaling before it, meaningful data with low variance may be just dropped, if you do not properly scale before to apply it.

However, suppose that you are using color histogram features. You want to apply PCA at the 0.999 variance retain in order to avoid unpredictable behaviours of your classifier in real world deployment. If you perform features scaling before to apply PCA, you are not deleting anymore components with less variance, thus the one more affected by "noise". In this case I would not apply features scaling before dimensionality reduction. I would apply it after PCA, and before to feed features in the NN, always for easing convergence.

What about if you have histogram features and shape features? In this case, I would scale the two bunch of features separately. I would apply dimensionality reduction on them separately, and then I would scale them all together.

When dimensionality reduction is involved, you should apply features scaling always after its application, while you have to apply it before depending on the type of data.

Just for completeness, in few certain practical cases I encountered, not scaling features at all gave me slightly better results.

The best advice I can give you is to try.

Answer 2

From my experience, feature scaling is not always required. It's mainly to bring your features down to a comparable range so that the feature space is not skewed along any one particular feature. This will speed up convergence.

Whether you do it or not before training neural networks would depend on the data you're dealing with. For example, I won't do it when my feature vectors are binary one-hot or refer to indices of words (in language modelling). Likewise, also not if they are already in a comparable range, such as feature 1 in (0, 1) and feature 2 in (-1, 3) (arbitrary choice of ranges). On the other hand, if one of my features is in the range (0, 200) and another in the range (0, 1), I would definitely do it. You'll find a very nice summary on this subject from Prof. Andrew Ng in this video about feature scaling for gradient descent which is very pertinent to your question.

And this topic seems to have been already discussed on Cross-validated where the chosen answer states:

It's simply a case of getting all your data on the same scale: if the scales for different features are wildly different, this can have a knock-on effect on your ability to learn (depending on what methods you're using to do it). Ensuring standardised feature values implicitly weights all features equally in their representation.

You can find more about that question here:

How and why do normalization and feature scaling work?