A common good practice in Machine Learning is to do feature normalization or data standardization of the predictor variables, that's it, center the data substracting the mean and normalize it dividing by the variance (or standard deviation too). For self containment and to my understanding we do this to achieve two main things:

  1. Avoid extra small model weights for the purpose of numerical stability.
  2. Ensure quick convergence of optimization algorithms like e.g. Conjugate Gradient so that the large magnitude of one predictor dimension w.r.t. the others doesn't lead to slow convergence.

We usually split the data into training, validation and testing sets. In the literature we usually see that to do feature normalization they take the mean and variance (or standard deviation) over the whole set of predictor variables. The big flaw I see here is that if you do that, you are in fact introducing future information into the training predictor variables namely the future information contained in the mean and variance.

Therefore, I do feature normalization over the training data and save the mean and variance. Then I apply feature normalization to the predictor variables of the validation and test data sets using the training mean and variances. Are there any fundamental flaws with this? can anyone recommend a better alternative?


4 Answers 4


Your approach is entirely correct. Although data transformations are often undervalued as "preprocessing", one cannot emphasize enough that transformations in order to optimize model performance can and should be treated as part of the model building process.

Reasoning: A model shall be applied on unseen data which is in general not available at the time the model is built. The validation process (including data splitting) simulates this. So in order to get a good estimate of the model quality (and generalization power) one needs to restrict the calculation of the normalization parameters (mean and variance) to the training set.

I can only guess why this is not always done in literature. One argument could be, that the calculation of mean and variance is not that sensitive to small data variations (but even this is only true if the basic sample size is large enough and the data is approximately normally distributed without extreme outliers).

  • $\begingroup$ I find this a bit confusing. The OP says he IS doing feature normalisation on validation and test data sets. Your reply first says that his approach is correct. Then you say "one needs to restrict the calculation of normalisation parameters to the training set" which is not what he is doing. So your response contradicts itself by telling him what he is doing is correct and then suggesting otherwise. What am I missing here? $\endgroup$
    – mahonya
    Commented Oct 13, 2014 at 8:16
  • 7
    $\begingroup$ What the OP does is described in his last paragraph and this is exactly what I have said. Of course normalization is applied to the test/validation-set if it has been applied to the training set. The important point is, that the parameters of this normalization have been calculated on the training data only and not on the whole set. Hope this helps. $\endgroup$
    – steffen
    Commented Oct 13, 2014 at 10:06
  • 1
    $\begingroup$ Ah, thanks a lot. I misunderstood your answer. I though you were suggesting the 'application' of normalisation to training set only, which was clearly not what you have suggested. $\endgroup$
    – mahonya
    Commented Oct 13, 2014 at 14:01
  • $\begingroup$ The important point is, that the parameters of this normalization have been calculated on the training data only and not on the whole set. This will cause an issue that is: The max(), min() of validation(or test) set will huge large than train set. For example the train set max min is 70.91 and -70.91, but the max min for the normalized validation set is 6642.14 and -3577.99. Before they normalization, they are 16.32 -0.94 16.07 -0.99. This is happening in my real data set. $\endgroup$
    – huang
    Commented Feb 27, 2021 at 16:17

Feature normalization is to make different features in the same scale. The scaling speeds up gradient descent by avoiding many extra iterations that are required when one or more features take on much larger values than the rest(Without scaling, the cost function that is visualized will show a great asymmetry).

I think it makes sense that use the mean and var from training set when test data come. Yet if the data size is huge, both training and validation sets can be approximately viewed as normal distribution, thus they roughly share the mean and var.

  • 2
    $\begingroup$ The reason why data normalization can speed up gradient descent, I guess, is that without normalization the rss has elliptical contours, so given fixed learning rate, it might need more iterations for gradient descent to converge. Whereas with scaling, rss has circle contours (symmetric), so gradient descent converges fast. Am I right? $\endgroup$
    – avocado
    Commented May 19, 2014 at 6:58

The methodology you have described is sound as others have said. You should perform the exact same transformation on your test set features as you do on features from your training set.

I think it's worth adding that another reason for feature normalization is to enhance the performance of certain processes that are sensitive to differences to the scale of certain variables. For example principal components analysis (PCA) aims to capture the greatest proportion of variance, and as a result will give more weight to variables that exhibit the largest variance if feature normalization is not performed initially.

  • $\begingroup$ Very good point! thank you for bringing it in. I recall from my studies always normalizing the input matrix before computing PCA. $\endgroup$
    – SkyWalker
    Commented May 21, 2016 at 18:07

Let me illustrate as to why we have to do normalization only on training data set.Say we extract features of different characteristics of apples like length,breath of the bounding box surrounding the apples,color,etc.,Let's say we normalize the feature length on the whole data and we have got mean length as 7 cm and variance as 1 cm.But if we just use train data we may get mean as 6 cm and variance as 1 cm.Which actually mean that there are large apples in the test data set which was actually not learn in the train data set.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.