A common good practice in Machine Learning is to do feature normalization or data standardization of the predictor variables, that's it, center the data substracting the mean and normalize it dividing by the variance (or standard deviation too). For self containment and to my understanding we do this to achieve two main things:

  1. Avoid extra small model weights for the purpose of numerical stability.
  2. Ensure quick convergence of optimization algorithms like e.g. Conjugate Gradient so that the large magnitude of one predictor dimension w.r.t. the others doesn't lead to slow convergence.

We usually split the data into training, validation and testing sets. In the literature we usually see that to do feature normalization they take the mean and variance (or standard deviation) over the whole set of predictor variables. The big flaw I see here is that if you do that, you are in fact introducing future information into the training predictor variables namely the future information contained in the mean and variance.

Therefore, I do feature normalization over the training data and save the mean and variance. Then I apply feature normalization to the predictor variables of the validation and test data sets using the training mean and variances. Are there any fundamental flaws with this? can anyone recommend a better alternative?


Your approach is entirely correct. Although data transformations are often undervalued as "preprocessing", one cannot emphasize enough that transformations in order to optimize model performance can and should be treated as part of the model building process.

Reasoning: A model shall be applied on unseen data which is in general not available at the time the model is built. The validation process (including data splitting) simulates this. So in order to get a good estimate of the model quality (and generalization power) one needs to restrict the calculation of the normalization parameters (mean and variance) to the training set.

I can only guess why this is not always done in literature. One argument could be, that the calculation of mean and variance is not that sensitive to small data variations (but even this is only true if the basic sample size is large enough and the data is approximately normally distributed without extreme outliers).

  • $\begingroup$ I find this a bit confusing. The OP says he IS doing feature normalisation on validation and test data sets. Your reply first says that his approach is correct. Then you say "one needs to restrict the calculation of normalisation parameters to the training set" which is not what he is doing. So your response contradicts itself by telling him what he is doing is correct and then suggesting otherwise. What am I missing here? $\endgroup$ – mahonya Oct 13 '14 at 8:16
  • 4
    $\begingroup$ What the OP does is described in his last paragraph and this is exactly what I have said. Of course normalization is applied to the test/validation-set if it has been applied to the training set. The important point is, that the parameters of this normalization have been calculated on the training data only and not on the whole set. Hope this helps. $\endgroup$ – steffen Oct 13 '14 at 10:06
  • $\begingroup$ Ah, thanks a lot. I misunderstood your answer. I though you were suggesting the 'application' of normalisation to training set only, which was clearly not what you have suggested. $\endgroup$ – mahonya Oct 13 '14 at 14:01

Feature normalization is to make different features in the same scale. The scaling speeds up gradient descent by avoiding many extra iterations that are required when one or more features take on much larger values than the rest(Without scaling, the cost function that is visualized will show a great asymmetry).

I think it makes sense that use the mean and var from training set when test data come. Yet if the data size is huge, both training and validation sets can be approximately viewed as normal distribution, thus they roughly share the mean and var.

  • 1
    $\begingroup$ The reason why data normalization can speed up gradient descent, I guess, is that without normalization the rss has elliptical contours, so given fixed learning rate, it might need more iterations for gradient descent to converge. Whereas with scaling, rss has circle contours (symmetric), so gradient descent converges fast. Am I right? $\endgroup$ – avocado May 19 '14 at 6:58

The methodology you have described is sound as others have said. You should perform the exact same transformation on your test set features as you do on features from your training set.

I think it's worth adding that another reason for feature normalization is to enhance the performance of certain processes that are sensitive to differences to the scale of certain variables. For example principal components analysis (PCA) aims to capture the greatest proportion of variance, and as a result will give more weight to variables that exhibit the largest variance if feature normalization is not performed initially.

  • $\begingroup$ Very good point! thank you for bringing it in. I recall from my studies always normalizing the input matrix before computing PCA. $\endgroup$ – SkyWalker May 21 '16 at 18:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.