# Data normalization and standardization in neural networks

I am trying to predict the outcome of a complex system using neural networks (ANN's). The outcome (dependent) values range between 0 and 10,000. The different input variables have different ranges. All the variables have roughly normal distributions.

I consider different options to scale the data before training. One option is to scale the input (independent) and output (dependent) variables to [0, 1] by computing cumulative distribution function using the mean and standard deviation values of each variable, independently. The problem with this method is that if I use the sigmoid activation function at the output, I will very likely miss extreme data, especially those not seen in the training set

Another option is to use a z-score. In that case I don't have the extreme data problem; however, I'm limited to a linear activation function at the output.

What are other accepted normalization techniques that are in use with ANN's? I tried to look for reviews on this topic, but failed to find anything useful.

• Z-scores normalisation is sometimes used but I have a funny feeling it may the another name for bayer's answer?? – user3484 Mar 1 '11 at 20:50
• It's the same except for the whitening part. – bayerj Mar 2 '11 at 9:37
• If you are predicting a value (as you are) rather than a probability (ie regression rather than classification) you should always use a linear output function. – seanv507 Jan 18 '16 at 7:54
• Rank-Gauss by Michael Jahrer. It is rank then make it gaussian. – user3226167 Aug 7 '19 at 6:59

A standard approach is to scale the inputs to have mean 0 and a variance of 1. Also linear decorrelation/whitening/pca helps a lot.

If you are interested in the tricks of the trade, I can recommend LeCun's efficient backprop paper.

• Of course one should never try to blindly normalize data if the data does not follow a (single) normal distribution. stats.stackexchange.com/a/816/4581 – user4581 Jan 12 '12 at 16:59
• With neural networks you have to. Otherwise, you will immediately saturate the hidden units, then their gradients will be near zero and no learning will be possible. It's not about modelling (neural networks don't assume any distribution in the input data), but about numerical issues. – bayerj Jan 17 '12 at 6:54
• I am quite confused. This link (machinelearningmastery.com/…) says that Standardization is useful when the algorithm you are using does make assumptions about your data having a Gaussian distribution (Not the case of the NN). Otherwise, it says that you should use Normalization. Can someone enlighten me ? – ihebiheb Jun 1 '18 at 18:20
• @ihebiheb If you're using parametric statistics, it's assumed the distribution of values in your data are normal/Gaussian. If they're not normal, you would need to transform them into normal, usually using logs. This question/answer does not seem to be about that. It is about scaling. Scaling does not necessarily change the shape of the distribution, but shifts its mean and scales its variance. Scaling, in the context of ANNs, is usually about helping each of many variables to carry the same weight by giving them all the same mean and variance. This is independent of normality. – mightypile Dec 5 '19 at 12:08

1- Min-max normalization retains the original distribution of scores except for a scaling factor and transforms all the scores into a common range [0, 1]. However, this method is not robust (i.e., the method is highly sensitive to outliers.

2- Standardization (Z-score normalization) The most commonly used technique, which is calculated using the arithmetic mean and standard deviation of the given data. However, both mean and standard deviation are sensitive to outliers, and this technique does not guarantee a common numerical range for the normalized scores. Moreover, if the input scores are not Gaussian distributed, this technique does not retain the input distribution at the output.

3- Median and MAD: The median and median absolute deviation (MAD) are insensitive to outliers and the points in the extreme tails of the distribution. therefore it is robust. However, this technique does not retain the input distribution and does not transform the scores into a common numerical range.

4- tanh-estimators: The tanh-estimators introduced by Hampel et al. are robust and highly efficient. The normalization is given by

where μGH and σGH are the mean and standard deviation estimates, respectively, of the genuine score distribution as given by Hampel estimators.

Therefore I recommend tanh-estimators.

• I'm having a hard time finding information on this tanh estimator. Can anyone shed some light on this? – Ben Ogorek Oct 6 '19 at 0:03
• Finally found a paper that does a good job describing tanh estimators for normalization: wjscheirer.com/papers/wjs_eccv2010_fusion.pdf. They do not sound viable as a general purpose normalization option. "[Tanh estimators] are far more complicated to compute, compared to the adaptive z-scores...The tail points for three different intervals from the median score of the distribution must be defined in an ad hoc manner. These parameters can be difficult to determine experimentally, and if chosen incorrectly, limit the effectiveness of tanh-estimators. " – Ben Ogorek Oct 6 '19 at 15:14

I gave a similar answer here When conducting multiple regression, when should you center your predictor variables & when should you standardize them? but thought it was sufficiently different context that an answer could go here.

There is a great usenet resource http://www.faqs.org/faqs/ai-faq/neural-nets/part2/section-16.html

It gives in simple terms some of the issues and considerations when one wants to normalize/standardize/rescale the data. As it treats the subject from a machine learning perspective, and as your question is ML, it could have some relevance.

• You should write the main points of each links here, so no additional "navigation" is needed – leoschet Oct 4 '18 at 21:35

You could do

• min-max normalization (Normalize inputs/targets to fall in the range [−1,1]), or
• mean-standard deviation normalization (Normalize inputs/targets to have zero mean and unity variance/standard deviation)

If you are working in python, sklearn has a method for doing this using different techniques in their preprocessing module (plus a nifty pipeline feature, with an example in their docs):

import sklearn

# Normalize X, shape (n_samples, n_features)
X_norm = sklearn.preprocessing.normalize(X)


Well, [0,1] is the standard approach. For Neural Networks, works best in the range 0-1. Min-Max scaling (or Normalization) is the approach to follow.

Now on the outliers, in most scenarios we have to clip those, as outliers are not common, you don't want outliers to affect your model (unless Anomaly detection is the problem that you are solving). You can clip it based on the Empirical rule of 68-95-99.7 or make a box plot, observe and accordingly clip it.

MinMax formula - (xi - min(x)) / (max(x) - min(x)) or can use sklearn.preprocessing.MinMaxScaler

"Accepted" is whatever works best for you -- then you accept it.

In my experience fitting a distribution from the Johnson family of distributions to each of the continuous features works well because the distributions are highly flexible and can transform most uni-modal features into standard normal distributions. It will help with multi-modal features as well, but point is it generally puts the features into the most desirable form possible (standard Gaussian-distributed data is ideal to work with -- it is compatible with, and sometimes optimal for, most every statistical/ML method available).

http://qualityamerica.com/LSS-Knowledge-Center/statisticalinference/johnson_distributions.php