Data normalization and standardization in neural networks

Question

I am trying to predict the outcome of a complex system using neural networks (ANN's). The outcome (dependent) values range between 0 and 10,000. The different input variables have different ranges. All the variables have roughly normal distributions.

I consider different options to scale the data before training. One option is to scale the input (independent) and output (dependent) variables to [0, 1] by computing cumulative distribution function using the mean and standard deviation values of each variable, independently. The problem with this method is that if I use the sigmoid activation function at the output, I will very likely miss extreme data, especially those not seen in the training set

Another option is to use a z-score. In that case I don't have the extreme data problem; however, I'm limited to a linear activation function at the output.

What are other accepted normalization techniques that are in use with ANN's? I tried to look for reviews on this topic, but failed to find anything useful.

Answer 1

A standard approach is to scale the inputs to have mean 0 and a variance of 1. Also linear decorrelation/whitening/pca helps a lot.

If you are interested in the tricks of the trade, I can recommend LeCun's efficient backprop paper.

Answer 2

1- Min-max normalization retains the original distribution of scores except for a scaling factor and transforms all the scores into a common range [0, 1]. However, this method is not robust (i.e., the method is highly sensitive to outliers.

2- Standardization (Z-score normalization) The most commonly used technique, which is calculated using the arithmetic mean and standard deviation of the given data. However, both mean and standard deviation are sensitive to outliers, and this technique does not guarantee a common numerical range for the normalized scores. Moreover, if the input scores are not Gaussian distributed, this technique does not retain the input distribution at the output.

3- Median and MAD: The median and median absolute deviation (MAD) are insensitive to outliers and the points in the extreme tails of the distribution. therefore it is robust. However, this technique does not retain the input distribution and does not transform the scores into a common numerical range.

4- tanh-estimators: The tanh-estimators introduced by Hampel et al. are robust and highly efficient. The normalization is given by

where μGH and σGH are the mean and standard deviation estimates, respectively, of the genuine score distribution as given by Hampel estimators.

Therefore I recommend tanh-estimators.

reference https://www.cs.ccu.edu.tw/~wylin/BA/Fusion_of_Biometrics_II.ppt

Answer 3

You could do

• min-max normalization (Normalize inputs/targets to fall in the range [−1,1]), or
• mean-standard deviation normalization (Normalize inputs/targets to have zero mean and unity variance/standard deviation)

Answer 4

Rank guass scaler is a scikit-learn style transformer that scales numeric variables to normal distributions. Its based on rank transformation. First step is to assign a linspace to the sorted features from 0..1, then apply the inverse of error function ErfInv to shape them like gaussians, then I substract the mean. Binary features are not touched with this trafo (eg. 1-hot ones).

This works usually much better than standard mean/std scaler or min/max.

Do checkout this link

Answer 5

If you are working in python, sklearn has a method for doing this using different techniques in their preprocessing module (plus a nifty pipeline feature, with an example in their docs):

import sklearn

# Normalize X, shape (n_samples, n_features)
X_norm = sklearn.preprocessing.normalize(X)

Answer 6

Well, [0,1] is the standard approach. For Neural Networks, works best in the range 0-1. Min-Max scaling (or Normalization) is the approach to follow.

Now on the outliers, in most scenarios we have to clip those, as outliers are not common, you don't want outliers to affect your model (unless Anomaly detection is the problem that you are solving). You can clip it based on the Empirical rule of 68-95-99.7 or make a box plot, observe and accordingly clip it.

MinMax formula - (xi - min(x)) / (max(x) - min(x)) or can use sklearn.preprocessing.MinMaxScaler

Answer 7

"Accepted" is whatever works best for you -- then you accept it.

In my experience fitting a distribution from the Johnson family of distributions to each of the continuous features works well because the distributions are highly flexible and can transform most uni-modal features into standard normal distributions. It will help with multi-modal features as well, but point is it generally puts the features into the most desirable form possible (standard Gaussian-distributed data is ideal to work with -- it is compatible with, and sometimes optimal for, most every statistical/ML method available).

http://qualityamerica.com/LSS-Knowledge-Center/statisticalinference/johnson_distributions.php