# Data normalization and standardization in neural networks

I am trying to predict the outcome of a complex system using neural networks (ANN's). The outcome (dependent) values range between 0 and 10,000. The different input variables have different ranges. All the variables have roughly normal distributions.

I consider different options to scale the data before training. One option is to scale the input (independent) and output (dependent) variables to [0, 1] by computing cumulative distribution function using the mean and standard deviation values of each variable, independently. The problem with this method is that if I use the sigmoid activation function at the output, I will very likely miss extreme data, especially those not seen in the training set

Another option is to use a z-score. In that case I don't have the extreme data problem; however, I'm limited to a linear activation function at the output.

What are other accepted normalization techniques that are in use with ANN's? I tried to look for reviews on this topic, but failed to find anything useful.

• Z-scores normalisation is sometimes used but I have a funny feeling it may the another name for bayer's answer??
– user3484
Mar 1, 2011 at 20:50
• It's the same except for the whitening part. Mar 2, 2011 at 9:37
• I gave a similar answer here stats.stackexchange.com/questions/29781/… but thought it was sufficiently different context that an answer could go here. There is a great usenet resource faqs.org/faqs/ai-faq/neural-nets/part2/section-16.html It gives in simple terms some of the issues and considerations when one wants to normalize/standardize/rescale the data. As it treats the subject from a machine learning perspective, and as your question is ML, it could have some relevance.
– Paul
Aug 6, 2014 at 15:07
• If you are predicting a value (as you are) rather than a probability (ie regression rather than classification) you should always use a linear output function. Jan 18, 2016 at 7:54
• Rank-Gauss by Michael Jahrer. It is rank then make it gaussian. Aug 7, 2019 at 6:59

A standard approach is to scale the inputs to have mean 0 and a variance of 1. Also linear decorrelation/whitening/pca helps a lot.

If you are interested in the tricks of the trade, I can recommend LeCun's efficient backprop paper.

• Of course one should never try to blindly normalize data if the data does not follow a (single) normal distribution. stats.stackexchange.com/a/816/4581 Jan 12, 2012 at 16:59
• With neural networks you have to. Otherwise, you will immediately saturate the hidden units, then their gradients will be near zero and no learning will be possible. It's not about modelling (neural networks don't assume any distribution in the input data), but about numerical issues. Jan 17, 2012 at 6:54
• I am quite confused. This link (machinelearningmastery.com/…) says that Standardization is useful when the algorithm you are using does make assumptions about your data having a Gaussian distribution (Not the case of the NN). Otherwise, it says that you should use Normalization. Can someone enlighten me ? Jun 1, 2018 at 18:20
• @ihebiheb If you're using parametric statistics, it's assumed the distribution of values in your data are normal/Gaussian. If they're not normal, you would need to transform them into normal, usually using logs. This question/answer does not seem to be about that. It is about scaling. Scaling does not necessarily change the shape of the distribution, but shifts its mean and scales its variance. Scaling, in the context of ANNs, is usually about helping each of many variables to carry the same weight by giving them all the same mean and variance. This is independent of normality. Dec 5, 2019 at 12:08

1- Min-max normalization retains the original distribution of scores except for a scaling factor and transforms all the scores into a common range [0, 1]. However, this method is not robust (i.e., the method is highly sensitive to outliers.

2- Standardization (Z-score normalization) The most commonly used technique, which is calculated using the arithmetic mean and standard deviation of the given data. However, both mean and standard deviation are sensitive to outliers, and this technique does not guarantee a common numerical range for the normalized scores. Moreover, if the input scores are not Gaussian distributed, this technique does not retain the input distribution at the output.

3- Median and MAD: The median and median absolute deviation (MAD) are insensitive to outliers and the points in the extreme tails of the distribution. therefore it is robust. However, this technique does not retain the input distribution and does not transform the scores into a common numerical range.

4- tanh-estimators: The tanh-estimators introduced by Hampel et al. are robust and highly efficient. The normalization is given by where μGH and σGH are the mean and standard deviation estimates, respectively, of the genuine score distribution as given by Hampel estimators.

Therefore I recommend tanh-estimators.

• I'm having a hard time finding information on this tanh estimator. Can anyone shed some light on this? Oct 6, 2019 at 0:03
• Finally found a paper that does a good job describing tanh estimators for normalization: wjscheirer.com/papers/wjs_eccv2010_fusion.pdf. They do not sound viable as a general purpose normalization option. "[Tanh estimators] are far more complicated to compute, compared to the adaptive z-scores...The tail points for three different intervals from the median score of the distribution must be defined in an ad hoc manner. These parameters can be difficult to determine experimentally, and if chosen incorrectly, limit the effectiveness of tanh-estimators. " Oct 6, 2019 at 15:14
• @BenOgorek and anyone wondering, check this out: stackoverflow.com/questions/43061120/… Sep 24, 2021 at 17:27

You could do

• min-max normalization (Normalize inputs/targets to fall in the range [−1,1]), or
• mean-standard deviation normalization (Normalize inputs/targets to have zero mean and unity variance/standard deviation)

Rank guass scaler is a scikit-learn style transformer that scales numeric variables to normal distributions. Its based on rank transformation. First step is to assign a linspace to the sorted features from 0..1, then apply the inverse of error function ErfInv to shape them like gaussians, then I substract the mean. Binary features are not touched with this trafo (eg. 1-hot ones).

This works usually much better than standard mean/std scaler or min/max.

If you are working in python, sklearn has a method for doing this using different techniques in their preprocessing module (plus a nifty pipeline feature, with an example in their docs):

import sklearn

# Normalize X, shape (n_samples, n_features)
X_norm = sklearn.preprocessing.normalize(X)


Well, [0,1] is the standard approach. For Neural Networks, works best in the range 0-1. Min-Max scaling (or Normalization) is the approach to follow.

Now on the outliers, in most scenarios we have to clip those, as outliers are not common, you don't want outliers to affect your model (unless Anomaly detection is the problem that you are solving). You can clip it based on the Empirical rule of 68-95-99.7 or make a box plot, observe and accordingly clip it.

MinMax formula - (xi - min(x)) / (max(x) - min(x)) or can use sklearn.preprocessing.MinMaxScaler

"Accepted" is whatever works best for you -- then you accept it.

In my experience fitting a distribution from the Johnson family of distributions to each of the continuous features works well because the distributions are highly flexible and can transform most uni-modal features into standard normal distributions. It will help with multi-modal features as well, but point is it generally puts the features into the most desirable form possible (standard Gaussian-distributed data is ideal to work with -- it is compatible with, and sometimes optimal for, most every statistical/ML method available).

http://qualityamerica.com/LSS-Knowledge-Center/statisticalinference/johnson_distributions.php